welcome

Thank you for purchasing the MEAP for Secure APIs. I hope you find this a useful resource and valuable companion in your efforts to protect your own APIs.

What does it take to protect an API? I faced this question many years ago when I started to build my first APIs. And the question stuck. The ensuing years were a fascinating journey of delving into the intricacies of API security. Is it API tokens? What kinds of tokens? How does Open Authorization work? How does it relate to OpenID Connect? How do you prevent SQL injection? How do you detect thresats in real-time? When do you know you’ve done enough to secure your APIs?

If these questions are boiling in your mind, you’ve come to the right place, or so I hope! The goal of this book is to answer all those questions and more. And we’re going to tackle those questions with practical examples. In my experience, it’s not enough to know what SQL injection is. The aha! moment comes when you see it in action and can correlate it with a specific bit of code. The same is true for all other API security exploits. This means you’ll find tons of coding examples in the book, and I highly encourage you to go through them in detail, run the code, and play around with it.

The book covers all the technical aspects of API security, including the OWASP top 10 API security vulnerabilities, design vulnerabilities, observability and threat detection, architecting secure APIs, authentication and authorization protocols, automated testing strategies, and more. But it doesn’t stop there. After many years of working in this field, I’ve realized that API security is everybody’s job, and implementing a robust API security program requires a strong alignment with your organization, and I dedicate specific sections of the book to explaining how to do that.

What about Generative AI (GenAI)? As I write, GenAI is in every conversation. Every client I work with is trying to figure out how to create a competitive edge with GenAI. And API security is no different. You’ll learn to leverage GenAI and Large Language Models (LLMs) to analyze your APIs, find vulnerabilities, and protect your APIs.

I’m very excited about this book and I look forward to hearing your feedback! Since this is a MEAP release, you have a unique opportunity to help me improve the book. Something doesn’t make sense? I forgot to explain a concept? You want more examples? Or a deeper dive into a certain subject? Leave your comments in the liveBook Discussion Forum and I’ll incorporate your feedback into the book.

Thank you!

— José Haro Peralta

In this book

welcome 1 What is API security? 2 Aligning API security with your organization 3 API security principles 4 Top API authentication and authorization vulnerabilities 5 Top API configuration and management vulnerabilities 6 API security by design 7 API authorization and authentication 8 Implementing API authentication and authorization 9 Secure API infrastructure 10 Financial-grade APIs 11 Observability for API security 12 Testing API security
Appendix A. API security checklist Appendix B. Setting up Auth0 for authentication and authorization Appendix C. API security RFCs and learning resources

1 What is API security?

This chapter covers

What is API security?

What is security-by-design for APIs?

Why does API security matter?

What are attack vectors in an API?

How does API security fit into the API development cycle?

How to keep pace with the field of API security

Application programming interfaces (APIs) are everywhere. Accounting for 57% to 83% of all Internet traffic (54% according to Cloudflare: https://www.cloudflare.com/2024-api-security-management-report/, 83% according to Akamai: https://www.akamai.com/newsroom/press-release/state-of-the-internet-security-retail-attacks-and-api-traffic), APIs are the engines that power the Internet. They allow organizations to offer services through well-defined interfaces. They power web- and mobile-based applications, IoT integrations, and more. They enable service-to-service communication. They accelerate automation and bring uniform interfaces to core platforms. More and more organizations are discovering the benefits of APIs to optimize their services, automate their processes, and tap into new lines of business.

But here’s the kicker: APIs represent gateways into our systems. Every time we create an API, we open a door that allows users to access data and functionality from our system. Of course, that’s the goal of an API: to expose data and functionality. And when properly implemented and secured, APIs are great. The danger comes when APIs aren’t properly secured.

Hackers take advantage of improperly secured APIs to break into our systems. This exposes our organizations to security breaches, violations of user privacy legislation such as Europe’s GDPR, and hefty fines in sectors like finance and healthcare. APIs are necessary to keep our systems up and running, but they can also put our business and reputation at stake.

Salt Security’s 2023 Q1 API Security Trends report found that API attacks grew by 400% in the first quarter of 2023. According to research by Akamai, in 2022 organizations suffered over 9 billion attack attempts, with some organizations facing over 10 million attacks per day (https://www.akamai.com/blog/security/attack-surface-workout-web-application-api-attacks). And API breaches aren’t cheap, with the cost of API breaches averaging $6.1 million, according to research by Kong (https://konghq.com/blog/enterprise/apis-are-mission-critical).

The obvious question is, how do we deal with API security? What do we need to do to ensure our APIs are secure? If these questions are bugging you, you’ve come to the right place.

In this chapter, you’ll learn what API security is and why it’s important. You’ll learn about the attack vectors in an API. As you’ll discover, API security isn’t just about authentication and authorization. It’s a much bigger world, and there are so many things to consider. You’ll learn what security-by-design is and how we apply it to APIs. Finally, you’ll learn how API security fits into the API development cycle, and how to keep up to date with the rapidly changing landscape of API security.

1.1 What is API security?

API security is the field that studies API vulnerabilities and how to detect and prevent them. It includes every part of our applications and systems that affect API security, such as authentication, authorization, user input manipulation, data validation, network configuration, observability, and so on. Figure 1.1 gives an overview of the field of API security. Take a moment to study the figure carefully, since it represents a mental model for the topic of this book.

Figure 1.1 The field of API security includes everything that can compromise the security of our APIs, including API design, implementation, architecture, and infrastructure. Every element plays a role in keeping our APIs safe and secure.

Definition

API security is the practice of designing, building, and operating of APIs securely to detect and prevent vulnerabilities such as unauthorized access, data leaks, and other exploits and misuses. Good API security takes a holistic approach and encompasses every element in the system that can compromise API security, including API design, user input manipulation, use of third-party APIs, libraries and dependencies, observability, network configuration, or database protection.

Good API security requires a holistic view of our APIs. As you can see in figure 1.2, every attack vector in an API is relevant for API security. It includes elements such as the API implementation, configuration management, and the infrastructure that operationalizes the API. For example, a crucial component of our infrastructure is network configuration. A common strategy to isolate components with different risk profiles in our system is network segmentation. You’ll learn more about networking and security in chapter 9, but long story short, segmentation is the practice of dividing out network into isolated subnets. How is this helpful? Take a look at figure 1.2. Figure 1.2 represents a website with two services –a user service and an admin service. The user service manages user data, while the admin service gives admin users full access to sensitive data and operations.

The user service’s API exposes an endpoint POST /user/avatar that takes a URL as input, and it downloads the user’s avatar from the provided URL. As illustrated in figure 1.2, the user service is vulnerable to server-side request forgery (SSRF) attacks, meaning a threat actor can exploit the service to trigger external calls to malicious websites, and can also send internal requests to other services, such as the admin service. In this case, the SSRF vulnerability provides an opportunity for privilege escalation –the threat actor uses the POST /user/avatar endpoint to send an internal request to the admin service, thereby gaining access to sensitive user data. There are a few vulnerabilities leading to this breach, including lack of input sanitization in the user service and lack of proper access controls between services. The one I want to emphasize here is lack of network segmentation, which allows the user service to talk to the admin service, thereby highlighting the importance of network configuration for our API security posture.

Figure 1.2 To secure our APIs, we first identify attack vectors, such as the API design and the database, then we model the possible threats, and finally, we harden our API by applying measures that prevent or mitigate those attacks.

As illustrated in figure 1.3, our job is to identify every part of the stack that can compromise API security, model the potential threats, and implement the right security measures. Here’s how it goes:

Identifying attack vectors: We look at every component of our system that can compromise our API security posture. We take a holistic view of the API. We look at the implementation, the infrastructure that operationalizes the API, the design, and authentication system, configuration management, and so on. Every component directly exposed to the user represents an attack vector.

Modelling API threats: Our job here is to identify what kinds of attacks can be performed. A good threat modelling exercise is explicit about the types of attacks, how they occur, and what the impact on the application and the business is. This analysis is crucial since it informs our security measures.

Securing APIs: In this step, we put the results of our previous analysis to work and address the vulnerabilities. For example, if we found vulnerabilities in our API design, we secure-proof the design. We ensure we handle user input carefully in every data entry point in the API. If we determine that the database is vulnerable, we place it in a private network and rotate its access credentials frequently. In the coming chapters, you’ll learn how to protect adequately each and every layer of your stack.

We will talk more about threat modelling in section 2.2, but to give you a teaser, let’s see how we threat model the scenario in figure 1.2. The analysis is illustrated in figure 1.3. Here’s how it works:

We identify input parameters that can be exploited for SSRF attacks. The task here is to make sure the implementation sanitizes such input.

We notice that internal calls between services are not authenticated, meaning services can talk to each other without any restrictions. The task here is to review whether all this communication is necessary and whether we can restrict which services can talk to each other.

We notice that all services run within the same network without segmentations, meaning all services can send internal calls to each other. The task here is to assess whether we can isolate highly sensitive services by placing them in a separate, isolated network.

Figure 1.3 To analyze attack vectors, we look at the three axes of API security: API design, implementation, and infrastructure. We ask ourselves what vulnerabilities we expose in each axis, what do they look like, and what their consequences are.

As you can see in figure 1.3, it helps to break down the analysis of attack vectors into three layers. We’ll call these layers the three axes of API security:

Design: Our API design can expose vulnerabilities, often in unexpected ways. For example, unbound strings open the door for large payload attacks; exposing integer IDs makes it easier to discover unknown resources by manipulating the identifier in the URL; and lack of proper pagination makes the system vulnerable to overloads. You’ll learn more about all these kinds of vulnerabilities in Chapter 6 and how to address them.

Implementation: Our implementation is the greatest source of vulnerabilities, so we must pay careful attention to the code and test it exhaustively. It helps to apply best practices and use robust patterns, such as Data Transfer Objects (DTOs), data validation for incoming and outgoing payloads, parametrize all database queries, and so on. You’ll learn how to tackle the implementation securely in chapters 3 and 12.

Infrastructure: Our infrastructure has a big impact on our security posture. It includes things like network topology, firewalls, data storage, computing resources, load balancers, API Gateways, and more. Our architectural solutions determine how our APIs will scale, whether sensitive data is accessible through public or private networks, and so on. You’ll learn more about architecting secure infrastructure for APIs in Chapter 9.

Note

The three axes of API security is a framework for analyzing vulnerabilities in our APIs. The three axes are: API design, implementation, and infrastructure. Each axis exposes unique vulnerabilities and we must analyze how malicious users can exploit them to cause damage to our system.

Besides these elements, there are a few other areas we must consider, especially in the area of operations, such as configuration and secrets management, rate-limiting, API versioning, logging and monitoring, threat detection, and so on. We call this approach shield right (as opposed to shift left), and you’ll learn more about this topic in chapter 11.

1.2 What is API security by design?

Security by design is a paradigm in cybersecurity that means our software is foundationally secure. As illustrated in figure 1.4, secure by design means we shift our security concerns to the left, to the beginning of the process, to the design stage. The goal is to shorten the feedback loop between the test and the implementation. As shown in figure 1.4, the traditional model means we have a very long feedback loop between the test and the implementation, which means we will not know if we are on the right track until it’s too late, and by that time we may have made some design or architectural decisions that are not easily reversible. By shifting our tests to the left, we shorten the feedback loop, which means we can proceed with more confidence in every step of the software development process.

Figure 1.4 In the traditional approach to API security, we wait until deployment to run our tests and discover vulnerabilities. We typically discover hundreds of vulnerabilities, and we have to go back to the beginning of the process multiple times to address them. With security by design, we address security in every step of the development cycle, and therefore we have fewer surprises when we run the tests after deployment.

Definition

API security by design is an approach that incorporates security concerns early in our development process, from the design stage, and throughout the entire API development cycle to ensure our implementation and delivery process are safe and secure. This is also known as shifting left on security.

The same approach is applicable to APIs. Before we build an API, we must design it, and at this stage, we have a great opportunity to start addressing vulnerabilities. However, secure by design does not apply only to API design. We want to ensure our approach to API design, implementation, and architectural solution is secure by design. What does this mean in practice? Let’s break down security by design across the three axes of API security.

1.2.1 Design

It's best practice to begin our API development cycle with a design stage. This is what we call API first, or more specifically, API design first, and it helps development teams and business stakeholders agree on how the API should work before any work begins on the producer or the consumer side. According to Postman’s 2023 State of the API Report (https://www.postman.com/state-of-api/api-first-strategies/), 75% of respondents agreed that API-first allows developers to deliver better APIs faster. API design first helps to bring alignment between software and business, and it should be done involving collaboration from product, engineering, and security functions to ensure the API delivers the desired business outcomes with an acceptable risk profile. Design-first also improves the development process as it allows client and server developers to have a clear idea of how the API is expected to work, and therefore both teams can work independently and in parallel (see section 1.4 of Jose Haro Peralta’s Microservice APIs [Manning, 2022] for more on the benefits of API-design first).

NOTE

There has been a lot of debate about what API-first means and what the best methods are to put it into practice. A commonly accepted definition of API-first is the notion of APIs as products and applying product development methods to API development. API-first teams involve all the stakeholders in the API design process, apply strict quality checks, standards, and produce detailed documentation. A closely related idea is that of API-design first, which emphasizes the importance of designing our APIs before we embark in their development. For more insight into this debate, check out David Biesack’s “API Design First is not API Design First” (February 15, 2023, API Design Matters, https://apidesignmatters.substack.com/p/api-design-first-is-not-api-design) and Daniel Kocot’s talk “API First… Not,” at the 2023 Platform Summit, 16-18 October 2023, Stockholm (https://youtu.be/aJsmK3XRjjY).

Beginning our API development cycle with design is also better from a security point of view. Starting with design gives us a chance to ensure our API is robust and secure before we even begin implementing it.

Help! My APIs do not begin with a design. Am I screwed?

Starting with a design is the best way to apply security by design to your API development process, but it is not the only way. Many organizations have legacy APIs that did not start with a design, and a significant number of those APIs are not even documented. Even today, in my experience, most organizations build APIs without ever going through a design process.

In such cases, the specification is often generated from code using a generator. For example, API development frameworks like Python’s FastAPI generate API documentation automatically from the code. Some organizations employ full-time API writers who look at the code and produce the corresponding API specification.

How do we apply security by design in these cases? If you don’t have an API specification, the first step is to get yourself one. I can’t emphasize this enough: you can’t do proper API security without a specification. Trying to protect your APIs without having specifications is like trying to protect a house without knowing where it is or what it looks like. Even a bad or inaccurate specification is better than nothing, and the testing techniques discussed in chapter 12, like fuzzing, will help you fill the gaps.

There are several methods for producing API specifications, such as using tools to generate specifications from code or hiring an API writer. If your API is consumed by a browser application, you can use tools like Burp to intercept traffic and produce the specification from traffic data. If you have large and complex APIs for which none of these approaches are feasible, engage a specialist service to get you sorted.

Once you have a specification, you’ll be able to take control of your security and apply all the methods and techniques discussed throughout this book to improve your security posture.

Why is this important? Many organizations wait until the API is deployed to a development/test environment to conduct an API security test. In other words, most organizations treat security as a runtime issue, that is as a software characteristic that can only be tested against a running application. This is a shortsighted view of security. As you will learn in this book, there is a lot that can be done from a design point of view to ensure our APIs are secure and reliable. Tackling security from the design stage allows us to shift-left our security approach and to deliver secure-by-design APIs. As you’ll see in chapter 2, this allows you to deliver more secure APIs, and also allows us to deliver faster and with more confidence, thereby reducing the cost of developing new features.

TIP

Many API vulnerabilities can be addressed in the design. When we design secure APIs, we find fewer vulnerabilities at runtime testing, which means we’ll be less likely to have to repeat the whole API development cycle to address them. So don’t wait until deployment to test security!

From a design point of view, we want to ensure that our endpoints expose clearly defined operations. We want to apply good API design patterns, such as pagination. We want to avoid exposing predictable resource IDs, such as incremental integers. We want to avoid flexible data schemas that make data validation difficult. We want to constrain user input as much as possible to reduce the chances of malicious code injection. You’ll learn more about this topic in Chapter 6.

1.2.2 Implementation

Once we have designed the API, it’s time to implement it. At this stage, it’s important to choose robust and stable libraries that make it easier to implement APIs. Most languages have API-specific frameworks, like FastAPI for Python (https://github.com/fastapi/fastapi), FeathersJS for Node (https://github.com/feathersjs/feathers), or Spring for Java (https://github.com/spring-projects/spring-framework). Good API frameworks support full JSON Schema semantics for granular data validation, generate valid OpenAPI specifications, allow us to validate request and response data, including URL query and path parameters and headers, and make it easy to add standard authentication and authorization layers.

From an implementation point of view, we want to ensure our code complies with the API specification accurately. To enforce compliance, we test the implementation against the specification. We call this approach contract-testing. A popular approach to contract-testing is using fuzz testers like Schemathesis (https://github.com/schemathesis/schemathesis/) and restler-fuzzer (https://github.com/microsoft/restler-fuzzer). As you will learn in Chapter 12, fuzz testers are tools that generate valid and invalid payloads and check how the API handles them.

TIP

Don’t reinvent the wheel! Use API frameworks for API development. API frameworks come with built-in functionality to streamline data validation and serialization, so you have less work to do. Apply a zero-trust approach and validate and sanitize all the data. And validate the implementation using fuzz testers!

Use well-established patterns to transfer and sanitize data across application layers. For example, Data Transfer Objects (DTOs) ensure we only transfer the data we need from the API layer to the business and the database layers, and from the database to other layers. It’s also important to parameterize all database queries to mitigate the risk of SQL injection. Finally, we want to apply a zero-trust approach and thoroughly validate all data regardless of its origin. As you’ll learn in section 3.2, zero-trust means validating every input in our API regardless of its origin.

1.2.3 Infrastructure

Our APIs run on infrastructure, and the configuration and architectural choices of that infrastructure have a strong impact on our API security posture. Infrastructure includes everything from network topology and server configuration to load balancers, reverse proxies, API Gateways, web application firewalls (WAFs), and so on.

Here we also have an opportunity to ensure our setup is secure by design. For example, we can configure our network to disable traffic between certain services when they don’t need to talk to each other, and to isolate certain components with high-risk profiles. We want to configure restrictive firewalls that only allow access from trusted origins. We must choose a deployment model that scales to meet the needs of our API consumers. We also need front-facing components, such as API gateways and web application firewalls (WAFs), that bring an additional layer of security, including protection against denial of service attacks. You will learn more about all this in chapter 9.

TIP

Secure infrastructure goes a long way to protect our APIs. For example, leverage secure network topologies to restrict access to sensitive resources, and use technologies like load balancers, API Gateways, and web application firewalls (WAFs) to handle incoming traffic securely.

It is crucial that we can replicate our infrastructure accurately across various environments, and infrastructure as code (IaC) is our ally here. We can leverage tools like CloudFormation if you deploy to AWS, Azure Resource Manager if you deploy to Azure, and Cloud Builder if you use Google Cloud, or more generic tools like Terraform if you deploy to multiple clouds and want to keep your IaC generic and decoupled from the implementation details of each provider. IaC allows you to automate your infrastructure using code, and you can audit and test such code using static analysis tools like tfsec (https://github.com/aquasecurity/tfsec) to ensure your infrastructure is secure and properly configured.

1.3 Why is API security important?

We have established what API security is, but, why is it important? APIs are gateways into our systems. That means APIs allow users to access data and functionality from our services. If our APIs are not properly protected, they can allow malicious users to access data and operations they shouldn’t have access to, bring our servers down, cause unwanted behaviors, and harm our business. In highly regulated sectors like finance, payments, and healthcare, APIs fall within the scope of compliance frameworks like PCI-DSS and HIPPA, and failing to comply adequately will bring sanctions and potential restrictions to our business.

According to SmartBear’s State of Software Quality | API 2023 report, API security is the number 2 concern in organizations, with businesses in finance, banking, insurance, defense, and other data sensitive industries ranking API security as their top concern (https://smartbear.com/state-of-software-quality/api/). SmartBear’s findings don’t come as a surprise, since API attacks are on the rise. Salt Security’s State of API Security Q1 2023 report found that API attacks grew by 400% during the first quarter of 2023 (https://salt.security/api-security-trends), and according to Gartner, APIs are set to become the most common attack vector on websites[1]. If you run APIs in production or are planning to use APIs, security must be one of your top priorities.

APIs often expose sensitive data and user flows, which makes them likely candidates for a security breach. On January 19, 2023, T-Mobile disclosed that personal identifiable information (PII) from 37 million customers had been stolen due to improper access controls and weak authorization checks on their APIs[2]. Security breaches like this can cost your organization dearly.

API security incidents can also cause damage in less obvious ways. Lack of rate-limiting or API metering allows hackers to scrape valuable content from your website, while the inability to detect malicious activity allows hackers to sabotage your business model. Sensitive business flows are also often easier to abuse through APIs. For example, a common threat for e-commerce websites is scalping, the practice of buying out the whole stock of a high-demand product to re-sell it at a higher price. We have seen variations of this threat everywhere. For example, UK’s Driving and Vehicle Standards Agency (DVSA) recently had to deal with malicious users buying out all the available slots for driving tests and re-selling them at a much higher price[3].

The problem is compounded by lack of proper visibility. Most organizations lack metrics about API usage, let alone real-time monitoring of user activity that can be used to flag malicious behavior. According to IBM’s Cost of Data Breach Report 2024, it takes organizations on average 258 days to identify and contain a security breach (https://newsroom.ibm.com/2024-07-30-ibm-report-escalating-data-breach-disruption-pushes-costs-to-new-highs). With APIs under constant attack, this isn’t acceptable anymore. We can and must do better, and this book will teach you how.

API security incidents not only cause financial harm to the business, but more importantly, they damage the reputation of your organization. If your customers cannot trust you to handle their data securely, they will eventually stop using your platform.

Despite the damage to our business, it is customers who ultimately pay the price. When customer data gets leaked or malicious users get access to their accounts, our customers suffer financial and personal repercussions. As a business, you have a duty to protect your customer’s data, and securing your APIs properly is part of this deal.

APIs are both a necessity and a liability. We need APIs to offer our technology services to the world, but they can also compromise our very business model. To ensure our APIs deliver value for the business without causing damage, we must secure them properly.

1.4 Unexpected vectors of attack

Why are API attacks so common and so difficult to tackle? We’ve been building APIs for decades. However, API security is still poorly understood. Compared to traditional websites, APIs offer new and often unexpected vectors of attack. For example, APIs use structured schemas to represent request and response bodies, and depending on their configuration and implementation, threat actors may be able to abuse them for data corruption and mass assignment attacks. Every input parameter used in an API, whether it goes in the URL, headers, or the request body, can be exploited for SQL injection, server-side request forgery (SSRF), and other types of attacks.

Figure 1.5 Every input parameter in an API represents an attack vector, including URLs, query and path parameters, request headers, and payloads. Unconstrained user input allows hackers to send malicious requests to our API to harm our system or cause unexpected behaviors.

As you can see in figure 1.5, every input parameter in our API is a potential attack vector, and as you’ll learn in chapter 6, lack of constraints in those parameters only make things worse. To understand why, let’s look at an example of pagination. Pagination is the ability to slice a collection of items into smaller chunks so that we can process them more conveniently and efficiently. We need pagination in endpoints that represent a collection of items, such as a product catalogue on an e-commerce website.

A good pagination pattern allows the user to choose how many items they want to see per page, which page they want to look at, how they want to sort the items, and how they want to filter them out. All these input parameters represent attack vectors. For example, when filtering items from the list, a threat actor may set the value of the filter to a SQL injection payload such as GET /products?filter=' OR 1=1;--. If the filter parameter’s value goes straight into the database query, it may nullify all other query conditions, potentially leading to data disclosure, broken object-level authorization (BOLA), and more.

Threat actors can exploit input parameters for other types of attacks such as big integer attacks. For example, when selecting the number of items per page, a sensible choice is 10 or 20, but a malicious user may request 100 million per page. Requesting a large number of items from our API will put enormous pressure on our database, and possibly bring our site down.

The purpose of API attacks is not always to create havoc in our databases or to bring our sites down. We can also exploit vulnerabilities to introduce unwanted behaviors in the system or to cause errors. For example, when paginating the product catalogue, a user may request page -20. In this scenario, the server may crash, or it may return a random collection of items to the user, hence opening the door to a data leak.

Another major attack vector is request payloads. We use request payloads to send data to the server over a POST, PUT, or PATCH request, typically to create or update a resource. A common vulnerability in request payloads is allowing the presence of additional properties. For example, a fintech API may have an endpoint to make and update payments. Typically, the payment request will undergo a few checks, such as ensuring your account has the necessary funds for the payment. However, if the request payload allows the presence of additional properties, a malicious user may be able to manipulate the state of their payment and bypass the checks. Surprisingly, this is a common vulnerability in fintech APIs.[4]

But it isn’t just user input. Authorization is also more challenging with APIs. We typically implement API authorization with stateless tokens, that is tokens that contain all the information we need to validate that the user has access to the API. How do we tell legit tokens from bad tokens? As you’ll learn in chapter 7, access tokens contain a special component called the signature, which tells us whether the token is valid. It’s crucial to use strong signing algorithms and ensure that the token signature is always correctly validated. Despite sounding obvious, many organizations fail to validate their token signatures correctly, including the likes of Auth0 (https://apisecurity.io/issue-81-vulnerabilities-microsoft-teams-auth0-smart-home-hubs/) and Microsoft (https://apisecurity.io/issue-115-vulnerabilities-solarwinds-ledger-outlook-new-plugin-jetbrains-ides/).

Role-based access control (RBAC) checks can be more challenging too. Most applications have a concept of user roles, such as admin and non-admin users. Some APIs use specific endpoints or parameters for role-based access. For example, an API may have admin endpoints under a URL path, such as example.com/admin, or an admin subdomain such as admin.example.com, while some APIs flag admin users by using special headers such as X-User: admin. None of these strategies is inherently wrong as long as we apply robust access authorization in those endpoints. However, a lot of APIs are built under the assumption that admin endpoints will only ever be accessed by legit users, which of course opens the door for admin access breaches.

Another type of vulnerability that has emerged in recent years is known as unrestricted access to sensitive business flows (https://owasp.org/API-Security/editions/2023/en/0xa6-unrestricted-access-to-sensitive-business-flows/). This vulnerability occurs when malicious users abuse our API to take advantage of our business model. For instance, many applications offer promotional codes and discounts when you sign up or when you invite a friend to join, and threat actors can sometimes exploit such features by creating fake accounts. This is known as promo fraud and affects many organizations, such as PayPal, who was forced to close 4.5 million fake accounts in Q4 2021, causing its stock value to slump by 25% (https://www.forbes.com/sites/jeffkauflin/2022/02/02/paypal-admits-45-million-accounts-were-illegitimate-as-fintechs-fraud-problem-grows/). APIs that are vulnerable to these types of business flow exploits can incur significant costs. In chapters 4 and 6, you’ll learn how to address this vulnerability to keep your business model protected.

1.5 How does API security fit into the API development cycle?

The typical API development process involves four main stages: design, implementation, testing, and operations. Let’s see what happens at each stage:

Design: during the design stage, we gather requirements, decide the API style we are going to use, and consolidate our design into a formal specification. As we work through the design, this is an excellent opportunity to threat-model our API and identify potential threats (you’ll learn more about threat modelling in section 2.2). It is also a good time to write down some unit tests to validate our implementation later on. It’s important to conduct this stage in close collaboration with the business to be aligned in our design choices and the risks we want to assume.

Implementation: once we have an API design, it’s time to build. In this stage, we write the code that implements the API. This is also the time to decide what type of infrastructure we need for API and to set up and configure it.

Testing: once we’ve built the API, it’s testing time. Ideally, we’ll have written unit tests before and during the implementation stage. After implementing the API, we want to run fuzz testers to ensure the implementation is accurate and behaves as expected, we want to run end-to-end tests to ensure the whole system works reliably, we want to test the integration with the API client, and so on.

Operations: once we are satisfied that the API works as expected, it’s time to deploy to production and monitor its behavior and performance. As part of our operations, we want to ensure we have detailed logs, monitor user activity, and have detailed visibility of all user flows and errors. As you will learn in chapter 11, good observability is critical to detect threats early and react to them on time.

How does security fit into this development cycle? The goal of API security-by-design is to shift left our API security concerns. We start thinking about security from the very first step of our API journey, and we approach every step of the API development process following best practices for security. Figure 1.6 shows how security by design fits within the API development. Let’s see how that works out in practice.

Figure 1.6 Security by design fits into every step of the API development cycle. It encourages us to approach design, implementation, testing, and operations from a security point of view.

During the design stage, we want to ensure our API design is secure. The idea is not to get lost in security details while we are designing the API, but to consider how our design impacts security. For example, this is the time to consider what kinds of parameters we’ll expose to our users, and how we will prevent threat actors from abusing input parameters for SQL injection, SSRF, and other kinds of attacks.

When designing operations and user flows, we also want to consider how they affect security. For example, if we design an e-commerce API, how do we prevent scalping? In a ride-sharing API that gives out discount coupons for every referral signup, how do we prevent malicious users from signing up fake users? In fintech APIs, how do we ensure that malicious users can’t bypass all the necessary checks before a transaction is executed? Our solutions to these problems have a direct impact on user experience, so it’s important to approach them in collaboration with our business stakeholders. Throughout the book, you’ll learn strategies to model these security threats and deal with them. As we mentioned earlier, the design stage is an excellent time to write down some unit tests to help us validate the implementation later.

Once we are done with design, it is time to move on to implementation. From a security-by-design perspective, we want to ensure we are using robust and secure frameworks and libraries and keep them up to date. Our implementation will deal with real data in a live environment, and it’s crucial to validate data thoroughly at every level of the stack. Applying the principle of zero-trust security, we validate and sanitize both incoming and outgoing data. It’s also good practice to parametrize all our queries against the database, and to use well-known patterns for secure data handling, such as Data Transfer Objects (DTOs). You’ll learn more about secure implementation strategies throughout the rest of the book.

As you work through the implementation, keep writing unit tests to ensure every layer of your code is working as intended. It’s also a good idea to run your code through static application security analysis (SAST) and software composition analysis (SCA) tools. SAST checks for common flaws, vulnerabilities, and misconfiguration errors in your code, while SCA detects known vulnerabilities (common vulnerabilities and exposures, or CVEs, https://www.cve.org/ and https://nvd.nist.gov/vuln) in your third-party dependencies. Some of the most popular tools in this space include Snyk (https://snyk.io/), Semgrep (https://semgrep.dev/), Aikido (https://www.aikido.dev/), checkmarx (https://checkmarx.com/), Renovate (https://github.com/renovatebot/renovate), and GitHub’s dependabot (https://github.com/dependabot).

After implementing the API, it’s time to test the hell out of it! We have several goals at this stage: we want to ensure our server implementation is fully compliant with the API design, that it exposes the right behavior, and that it’s secure and robust. We use tools like fuzz testers to validate the server implementation against the API specification, and API security testing tools to ensure the implementation is secure. It’s also a good idea to implement a suit of integration tests that validate the expected API behavior and simulate user flows.

Finally, it’s time to deploy the API and operate it in the wild. We want to ensure our Continuous Integration and Delivery pipeline tests our APIs thoroughly and prevents us from releasing insecure APIs. We want to have world-class API monitoring, observability, and alerting in place. We want to be able to tell how our APIs are being used, and to detect and block malicious behavior as soon as they show up.

1.6 The rapidly changing landscape of API security

Now that we understand what security-by-design is and how we apply it at every stage of the API development cycle, let’s consider the current and future state of API security.

API security is a rapidly evolving field. We hear about security incidents nearly every other week, and the rate of growth of API breaches is only accelerating. APIs are becoming the most likely vector of attack in our systems, and things are going to become worse. Why? There are four factors contributing to this trend: the rapid growth of APIs, the growing complexities of APIs, the growing number of connected devices, and more recently, the addition of Generative AI. Let’s examine the role of each factor.

The number of APIs available on the Internet is growing very rapidly. Nearly every organization with an in-house technical system uses APIs. We use APIs to automate internal processes, to drive integrations between microservices, to enable integrations with browser and mobile applications, and to deliver products. According to Noname’s 2022 The API Security Trends Report (https://nonamesecurity.com/resources/api-security-trends-report), large organizations use over 25,000 APIs on average, and that number is likely to continue growing. The rapid growth of APIs makes it increasingly challenging to manage them, to roll out security improvements, and to have adequate visibility of all the API activity. This, in turn, makes API security more difficult.

But it’s not just the growth in the number of APIs. APIs are also becoming more complex! Modern web applications represent complex user flows such as booking a holiday, applying for a mortgage, filing a return, or real-time collaboration in a shared document. When we design APIs, we must break down complex user flows into a series of independent, stateless steps. Flaws in the design open the door for abuse and manipulation of our business model.

For example, when applying for a mortgage, the lender runs a risk assessment against the applicant, checking things like their income, employment, civil status, number of dependents, property valuation, credit rating, and so on. Some of these checks may need to happen in a sequential order and they may depend on each other. If we fail to model user flows and input accurately and securely, malicious users will be able to skip checks and get their applications approved when they shouldn’t.

While user-facing APIs face a growing number of security challenges, the Internet of Things (IoT) presents an even more fertile ground for security incidents. When we connect something to the Internet, we increase its attack surface and often becomes more easily hackable. Hence IoT offers unique opportunities to hijack and steal smart cars, break into homes, and steal from stores[5]. Even a traditional laundry business becomes hackable when the washing machines connect to the Internet, a revealed by Alexander Sherbrooke and Iakov Taranenko, former students at UC Santa Cruz, who found a vulnerability in the University’s laundry service API that allowed them to use the service for free (https://techcrunch.com/2024/05/17/csc-serviceworks-free-laundry-million-machines/).

The number of smart devices connected to APIs is growing rapidly with the rise of smart homes, smart stores, smart cities, and so on. Protecting IoT devices is challenging. Some devices ship with hardcoded API keys, so anyone who gets their hands on the device can steal the key and get access to the API. Malicious users may also be able to spoof the connection between devices and APIs, therefore gaining access to sensitive information. Finally, connected devices are also vulnerable to hijacking attacks, meaning malicious users can take control of them and gain access to private networks or repurpose their functionality to launch DDoS attacks and more[6].

Finally, the rise of generative AI is poised to make API security even more challenging. Many applications use face or voice recognition to perform access security checks, and the widespread availability of deepfakes makes those checks increasingly vulnerable[7]. With the raise of Generative AI, it’s never been easier to hack websites. As we’ll see throughout the book, malicious users can leverage Large Language Models (LLMs) like ChatGPT to generate hacking strategies, such as SQL injection attacks, and to put together the code necessary to execute the attacks. We can also expect models specialized on cybersecurity to become available soon, making it even easier to launch attack against websites[8].

More and more developers are using Generative AI as a coding assistant in the daily development job, either to understand errors, to get coding examples, or to generate full code snippets. The use of Generative AI in software development can boost developer productivity, but it can also make web applications vulnerable if the output from AI models isn’t thoroughly checked, tested, and corrected. Throughout the book, we’ll learn to tackle these challenges.

1.7 Who is this book for and what you will learn

APIs are ubiquitous on the Internet. Nearly every system consumes external services such as payments processing, geolocation, and emailing over APIs. And a growing number of applications expose APIs for various purposes such as driving integrations between microservices, or for consumption by mobile and browser applications. All these scenarios pose security threats, both for consumers and for producers. This book is for everyone who sits on either side of an API.

If you’re consuming APIs, this book is for you. If you’re exposing APIs, even if those are so-called “internal” APIs, this book is for you. If you architect distributed systems that rely on APIs, this book is for you. If your organization delivers products and services through APIs, this book is definitely for you.

As part of my job, I get to talk to organizations about API security, and I get the opportunity to interview individuals of all levels, from junior developers to CTOs. My experience is that API security is a poorly understood topic. This isn’t surprising, since as saw earlier, APIs expose attack vectors in very unexpected ways, and the traditional recipes for application security are insufficient for APIs. My hope is this book will help to address this skills gap, raise awareness about the risks associated with APIs, and foster better security practices in API development.

This book teaches you what the main types of security API vulnerabilities are, and crucially, how to protect your APIs against them. You’ll learn to

Apply principles and best practices of API security

Tackle the main types of API security vulnerabilities

Apply security by design to your APIs

Identify and address vulnerable API design

Automate the process of identifying and tackling vulnerabilities in your APIs

Implement APIs following best security practices

Design secure infrastructure for APIs

Apply robust access authorization controls to your APIs

Implement financial-grade API security

Operate APIs securely

Leverage API observability to detect malicious user activity and react to it

By the end of this book, you will be aware of all the main types of vulnerabilities that APIs face and how to tackle them. You will know how to build APIs that are secure by design. You will know how take into account security considerations when you design APIs and when you model user interactions. You will know how to implement and architect APIs securely. You will know how to automate the process of identifying and tackling security vulnerabilities in your APIs. As we saw at the beginning of the chapter, between an estimated 57-83% of all Internet traffic goes through APIs, and so my hope is that this book will help software developers across the world to build a more secure Internet.

1.8 Summary

API security incidents are very costly for organizations. Security breaches can incur big penalties, while exposing vulnerable user flows allows hackers to abuse our business model, resulting in a potential loss of income and loss of reputation.

Every input parameter in an API represents an attack vector, including URL query and path parameters, request payloads, and headers.

Security by design is an approach that encourages us to shift left our API security strategy. With security by design, we apply security considerations from the design stage of our API development process.

We can tackle many security vulnerabilities in the design, for example by exposing safe user flows, constraining user input, and by not exposing server-side properties in user input.

Security by design encourages us to use secure implementation strategies, such as parametrizing all database queries and avoiding mass assignment.

We must continuously test our APIs to detect vulnerabilities, and we must automate as much of the testing process as possible.

Secure architectural solutions such as deploying databases to private networks, using robust API gateways, and enforcing strict firewall policies, play a major role in API security.

We must constantly monitor our APIs and run real time analysis of user activity to detect malicious behavior and react to it timely.

The growing number of APIs and connected IoT devices, the increasing complexity of APIs, and the emergence of Generative AI present unique new challenges in the current space of API security.

[1] Victor Dey, “Why API security is a fast-growing threat to data-driven enterprises,” VentureBeat, November 23, 2022 (https://venturebeat.com/security/why-api-security-is-a-fast-growing-threat-to-data-driven-enterprises/).

[2] Tim Keary, “T-Mobile data breach shows API security can’t be ignored,” VentureBeat, January 20, 2023 (https://venturebeat.com/security/t-mobile-data-breach-shows-api-security-cant-be-ignored/).

[3] Jon Ungoed-Thomas, “An absolute mess’: learner drivers forced to buy tests on black market as companies block-book slots,” The Guardian, September 3, 2023 (https://www.theguardian.com/uk-news/2023/sep/03/an-absolute-mess-learner-drivers-forced-to-buy-tests-on-black-market-as-companies-block-book-slots).

[4] I discuss an example of this vulnerability in my talk “API Security by Design,” at the Open Security Summit, January 16, 2024 (https://youtu.be/gJinDI_Ma1Y, especially from minute 28:41).

[5] Nic Surpatanu, “The Connected Car is the Next Attack Vector,” Forbes, September 7, 2022 (https://www.forbes.com/sites/tanium/2022/09/07/the-connected-car-is-the-next-attack-vector/).

[6] Pooja Kumari and Ankit Kumar Jain, ”A comprehensive study of DDoS attacks over IoT network and their countermeasures,” Computers & Security, 127 (April 2023) [available online: https://www.sciencedirect.com/science/article/abs/pii/S0167404823000068].

[7] Timur Yunusov, “Synthetic identities – an AppSec point of view,” presentation at OWASP’s London Chapter Meetup, February 28, 2023 (recording available online: https://youtu.be/vQZayZV_C90).

[8] Bill Doerrfeld, “Why Generative AI is a Threat to API Security,” Security Boulevard, July 20, 2023 (https://securityboulevard.com/2023/07/why-generative-ai-is-a-threat-to-api-security/).

2 Aligning API security with your organization

This chapter covers

Evaluating your API security posture

Modeling threats for your APIs

Kicking off your API security journey with low hanging fruits

Creating an API security program for your organization

Getting buy-in from your organization to tackle API security

Navigating API security audits

As we saw in chapter 1, APIs are becoming the main vector of attack on the Internet. If you work or are planning to work with APIs, it’s crucial to start thinking about security as early as you can. The questions are, how do you factor security into your API development? How do you align security with your product goals and requirements? How do you implement continuous security checks as part of your API development process?

Software development is a social activity. We build software as part of a team, which is part of a bigger organization. Building software involves talking with different stakeholders, understanding product requirements, prioritizing features and concerns, and working together towards a common solution. We must factor in deadlines, which means not all features get the same attention or get to be developed at the same time.

For example, the style and performance of a user interface (UI) have a direct impact on the user experience (UX), so we prioritize them. But what about something like API security? API security also has a direct impact on the user, but it’s not as obvious, at least not until you get a data breach.

We can talk about the technical aspects of API security all we want, but if we don’t address its social aspects, we may fail to align security with our product goals. How do we make the case for API security? How do we ensure our stakeholders understand its importance and allocate time to work on it? How do we align our organization with a sensible API security strategy?

In this chapter, you’ll learn strategies to align API security with our organization’s goals to ensure it’s adequately prioritized. The first step is to evaluate our API security posture to understand where we stand and what we must tackle first. You’ll learn to model threats for your APIs to understand your risks, and how to implement an API security program that aligns with your organization’s goals.

2.1 Evaluating your API security posture

Your security posture is your ability to prevent, identify, respond to, and recover from security threats. To determine how good your security posture is, you must evaluate it. In my experience, many organizations don’t evaluate their security posture until they go through a major event, such as a round of investment with its related due diligence, or a regulatory audit. I recommend you don’t wait for such events and start now. As we saw in chapter 1, a data breach has damaging consequences for your organization. It’s in your best interest to know how well prepared you are to deal with such security incidents, and it all starts with a security posture evaluation.

There are different frameworks to help you evaluate your security posture, like NIST SP 800-53 (https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final), CIS (https://www.cisecurity.org/controls), and ISO/IEC 27001 (https://www.iso.org/standard/27001), among others.[1] Those frameworks provide very comprehensive and detailed guidance to help secure your organization, and only part of it is relevant to API security. Figure 2.1 maps the most common requirements in cybersecurity frameworks to API security.

Figure 2.1 Mapping of the most common requirements in cybersecurity frameworks for API security.

Let’s break down each category in figure 2.1 one by one with questions you need to answer for each category:

Data inventory: What kind of data do you collect? How do you collect it? Where do you store it? For how long? Do you have an inventory and is it regularly updated? How does this data get exposed through the API? What are the most sensitive data endpoints?

Attack surface: How many endpoints do you expose? How many different versions of the API do you have? Do you have processes in place to prevent exposure of unknown attack surface?

Authentication and authorization: Do you use best practices and standards for authentication and authorization? Are your login endpoints adequately protected from brute force and other forms of attacks? Do you issue access tokens with robust signatures? Do your access tokens contain sensitive information? Do you rotate your signing keys often?

Access controls: Are all your entry points protected? Do you have strict access controls for sensitive endpoints? Are access policies adequately enforced? Who in your organization has access to sensitive assets and how? Are access controls correctly implemented?

Secure implementation: Do you use secure implementation patterns? Do you use proper API libraries to handle data validation? Are your dependencies secure? Is your API design and architecture secure? Do you review and test your code before release?

Security testing: Do you run automated security tests against your APIs regularly? Have you ever run a specialized penetration test against your APIs?

Risk analysis: What are the main security risks for your organization? When given a list of vulnerabilities, do you know how to prioritize them?

Incident detection: Do you have API observability? Do you constantly analyze API activity to detect malicious activity? Do you receive alerts when a threat is detected?

Incident response plan: Do you have a detailed response plan for security incidents? Are some of your incident responses automated? Does your response plan comply with your regulatory requirements?

Training and awareness: Are all company employees aware of the risks associated with your APIs? Do you provide ongoing training to keep them up to date with the latest threats?

Your answers to the above questions determine your level of API security readiness and maturity. If you can’t answer any of the questions, your API security readiness is low. For example, if you don’t have a data inventory or haven’t mapped your attack surface, you have a low readiness in those areas. To evaluate your API security posture, you must tackle your API security readiness first. Once you can answer the questions, you can evaluate your maturity. For example, having logs and software telemetry is a good start, but if you’re not actively monitoring those logs to detect malicious activity, your maturity in this space is low. Table 2.1 outlines a simple and actionable way to assess your API security posture.

Table 2.1 API security posture evaluation questionnaire. Answer “Yes” only when you’re 100% into a category. For example, if you’ve mapped 95% of your attack surface, that’s still “No”, and if your data inventory is updated regularly but not continuously on every change, that’s still “No”.

Mature

Not Mature

Data Inventory:

Is your data inventory up to date and continuously updated?

Yes

No

Attack surface:

Is your attack surface completely mapped and always up to date?

Yes

No

Authentication and authorization:

Do you follow best practices and standards for authentication and authorization?

Yes

No

Access controls:

Do you have secure and strict access controls and are they continuously updated?

Yes

No

Secure implementation:

Do you use secure design and implementation patterns?

Yes

No

Security testing:

Do you continuously test your APIs for security?

Yes

No

Risk analysis:

Have you mapped out your API business risks and are they continuously updated?

Yes

No

Incident detection:

Do you have proper API observability and automated detection of malicious activity?

Yes

No

Incident response plan:

Do you have a detailed response plan for security breaches and is it continuously updated?

Yes

No

Training and awareness:

Do you continuously train your staff on API security?

Yes

No

Table 2.1 provides a simple framework for API security posture assessment. The answers to these questions are binary, so following that framework, you’ll be either mature or not mature. If you want a more nuanced approach, choose a score from 0 to 5 in each category. For example, you may currently have an up-to-date data inventory, but instead of updating it when your data profile changes, maybe it gets updated once a year, monthly, or weekly. You can assign a more meaningful score for your specific situation and prioritize your security improvements based on your business needs. This is an introspection exercise. The only way to make this exercise useful is by being honest with your answers and scores. Use this framework to help you shape your roadmap towards a robust API security posture, and feel free to adapt it to your specific needs.

One word of caution regarding API security posture: many vendors in this space claim to be able to automate API security posture management. While some automation is certainly possible in this space, not all can be automated. Security posture management involves things like training, defining business risks, writing up incident response plans, and so on. You can’t automate your way out of those things. Even when it comes to purely technical security testing, not everything can be automated. As Corey Ball, author of Hacking APIs, highlights, most automated API security testing tools can’t detect most problems even in deliberately vulnerable APIs, and in many cases detecting vulnerabilities requires knowledge of the business domain, which many automation tools can’t capture. [2] So be wary of vendors who claim to be able to identify all security vulnerabilities in your APIs.

A fundamental component of assessing your API security posture is threat modeling. Threat modeling helps you understand your security risks, map your attack surface, and prepare your incident response plan. In the following section, you’ll learn to model API threats.

2.2 Threat modeling is a team sport

Threat modeling is an analysis of the security characteristics of your system. The goal of threat modeling is to identify vulnerabilities and how they can be exploited by threat actors. The idea is to think of different scenarios or threat models that represent how a malicious user can achieve their goals, such as breaking into a database, gaining unauthorized access to resources and operations, elevating their privileges, or injecting malicious content.

Threat modeling is a very useful exercise to raise awareness about system vulnerabilities. Even a simple threat model, as represented in figure 2.2, provides a lot of insight into how threat actors can break into your system, and what we need to do to secure it and detect the attacks. Figure 2.2 models a SQL injection attack that bypasses user access controls on a listing endpoint. As the model highlights, the attack happens because of unconstrained query parameters and because database queries aren’t parametrized. This is the actionable feedback from the model: we must constrain user input and parametrize the database queries.

Figure 2.2 A malicious user bypasses all access controls on the GET /products endpoint by exploiting a SQL injection vulnerability on the API.

Good threat models show how an attack goes through our system and which components are affected. For example, the model in figure 2.2 represents an attack that involves an API and a database. More complex systems may include API gateways, load balancers, multiple services and databases, queues and streams, and more components. Normally, each part of the system is owned by a different team, so good threat models have input from all teams. Effective threat modeling is a collaborative effort that brings together different teams, including engineers, product owners, quality assurance (QA) testers, business stakeholders, and others to get a holistic view of our system.

The question is, how do we go about modeling threats? To push for best practices in threat modeling, a group of industry leaders created the Threat Modeling Manifesto (https://www.threatmodelingmanifesto.org/). The manifesto defines the core values and principles of threat modeling and the main questions that every threat model must answer:

What are we working on?

What can go wrong?

What are we going to do about it?

Did we do a good enough job?

As you can see in figure 2.3, each question represents a step in the threat modeling process. To help us implement this process, the Open Worldwide Application Security Project (OWASP) has put together a useful guide to threat modeling (https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html). The guide breaks threat modeling down into four steps, each of which maps to a question in the Threat Modeling Manifesto:

application decomposition (what are we working on?),

threat identification and ranking (what can go wrong?),

response and mitigations (what are we going to do about it?), and

review and validation (did we do a good enough job?).

Let’s go through each step in detail.

Figure 2.3 Each question in the Threat Modeling Manifesto maps to a step in the threat modeling process. In this example, we show how we model a SQL injection vulnerability in a listing endpoint and how we go about fixing it.

2.2.1 Application decomposition

The first step is application decomposition, which maps to the question “What are we working on?” The idea is to understand how data flows through the system and what components are involved in processing data. Since we’re focusing on (malicious) user interactions, a helpful approach is using data flow diagrams (DFDs). Good DFDs help us visualize how data flows through the system and the components involved. For example, what happens when a user updates a resource through the API? Each component involved in processing data represents a potential attack vector. Figure 2.4 shows a DFD of a user updating their profile with their profile picture (avatar) with an external URL, highlighting in red the potential attack vectors.

Figure 2.4 DFD of a user updating their avatar using an external URL. In this case, the URL can be used to run a port scanning within our private network, access internal services, our databases, leak access tokens, and more.

A common mistake when modeling data flows is to overlook API calls to internal and external services. For example, in figure 2.4, a critical element of our system is the flow from the public API server to the internal API. If the public API is vulnerable to server-side request forgery (SSRF), our threat model must account for the fact that threat actors may be able to leak data from our internal API. However, we may fail to model such threat if the data flow between the public and the internal API is missing from our diagrams. To obtain an accurate representation of your data flows, ensure all relevant engineers and product stakeholders collaborate in the modeling and diagramming exercise.

2.2.2 Threat identification and ranking

The next step is threat identification and ranking, which maps to the question “What can go wrong?” In this step, we try to get into a hacker’s mindset and think of ways to break into or abuse the system. To help us model attack strategies, we use a threat modeling framework. The most popular threat modeling framework is STRIDE. Let’s see how threat modeling with STRIDE works and how it helps us identify threats.

NOTE

The STRIDE framework was developed by Loren Kohnfelder and Garg Praerit in 1999 while at Microsoft, in a paper titled “The Threats to our Products.” The paper is no longer available on Microsoft’s website, but you can download it from Adam Shostack’s website using the following URL: https://shostack.org/files/microsoft/The-Threats-To-Our-Products.docx. Adam is also the author of an excellent book on threat modeling titled Threat Modeling: Designing for Security (Wiley, 2014). Alternative frameworks to STRIDE include LINDDUN, OCTAVE, PASTA, TRIKE, and VAST. You can read more about them in OWASP’s Threat Modeling Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html).

Figure 2.5 shows you how STRIDE’s threat categories attack the system from every angle. Let’s break it down:

Spoofing: risk of stealing user credentials and taking over other user accounts.

Tampering: risk of corrupting or performing unintended updates on our data.

Repudiation[3]: risk of not being able to prove whether a user undertook a certain action.

Information Disclosure: risk of leaking system details, configuration, or sensitive data (data breach).

Denial of Service: risk of making the system unavailable.

Elevation of Privileges: risk of an attacker assuming privileges they don’t have.

Figure 2.5 Each threat category in STRIDE represents a different strategy to break into our system or steal data from our APIs.

Our job is to identify what can go wrong in each of these categories. For example, how can malicious users steal access tokens? How can they bypass access controls? How can they access data from other users? How can they steal configuration details of our system? Run brainstorming sessions with your team to come up with potential threats or play games such as Elevation of Privilege to help you come up with ideas if you are just getting started with threat modeling.[4]

Getting help with threat modeling

If you’ve never done threat modeling before, you may find it difficult to run your first threat modeling exercise within your organization. Like every other skill, threat modeling gets better with practice, and the more you do it, the better and more effective threat models you’ll get. If your organization is just getting started with threat modeling, start small and run threat modeling sessions frequently, for example every month or once a fortnight.

To help you get started with threat modeling, you may find it valuable to use popular tools like Microsoft’s threat modeling tool (https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool) or OWASP’s threat dragon (https://owasp.org/www-project-threat-dragon/). In these tools, you begin by representing your application’s data flows, and then the tools generate threat models based on those diagrams. Also, with the rise of Generative AI tools to help in software development, a host of applications are emerging in the space of cybersecurity, such as Matt Adam’s STRIDE GPT (https://github.com/mrwadams/stride-gpt), which you may find useful to get started generating threat models. When using tools like Microsoft’s threat modeling tool, OWASP’s threat dragon, and STRIDE GPT, you should review the threat models produced by the tools and add any changes you deem necessary.

Once you start gaining confidence with threat modeling, consider using tools like pytm (https://github.com/OWASP/pytm), which allows you to represent your data flows as code and generate threat models from them. Diagrams as code are great since they are more specific, accurate, and unambiguous, and can be more easily reviewed through pull requests.

A great resource to learn about threat modeling is Adam Shostack’s classic Threat Modeling: Designing for Security (Wiley, 2014). A great resource to learn to use OWASP’s threat dragon is Derek Fisher’s “Threat modeling with OWASP Threat Dragon” (https://youtu.be/mL5G8HeI8zI). For an overview of how STRIDE GPT works and what you can do with it, check out Matt Adam’s presentation “AI-Driven Threat Modelling with STRIDE GPT” at the Open Security Summit (Jan 15, 2024) (https://open-security-summit.org/sessions/2024/mini-summits/jan/threat-modeling/ai-driven-threat-modelling-with-stride-gpt/).

Check out also MITRE’s adversarial tactics, techniques, and common knowledge (MITRE ATT&CK, https://attack.mitre.org/) framework for a comprehensive repository of threat scenarios that you may find relevant to your applications.

The best way to answer the above questions is by representing the user and data flows involved in each threat and analyzing system vulnerabilities. For example, many organizations implement user management systems with custom authorization flows. Figure 2.6 represents a privilege escalation threat achieved due to a common vulnerability in such systems that allows malicious users to obtain access tokens from other users by cracking weak signatures and producing their own tokens.[5]

Figure 2.6 A common vulnerability in custom authorization servers is issuing tokens with weak signatures. In this example, a malicious actor manages to break the token’s signature and produce a token with admin privileges.

With the threats to our system identified, it’s time to rank them by priority. We rank threats by their likelihood of happening and their impact on the business. The ranking criteria are highly contextual to the nature of our organization and its business. For example, SQL injection attacks may be used to steal or corrupt data from our system. If our API design is vulnerable to SQL Injection attacks, that’s a red flag. However, if we parametrize all our database queries and use a Web Application Firewall (WAF, we’ll learn more about this in chapter 8), then SQL Injection is a low-likelihood threat. The prioritization work should be done in close collaboration with our stakeholders, especially legal and compliance teams, and it’s a great opportunity to raise awareness about API security risks with them.

2.2.3 Response and mitigations

Once we’ve identified and ranked our API threats, it’s time to answer the question “what are we going to do about it?” We must define a response to each threat. Adam Shostack, author of the seminal book Threat Modeling: Designing for Security (Wiley, 2014), distinguishes the following types of responses:

Mitigate: harden the system to reduce the likelihood of the threat. For example, implement rate-limiting in the API to prevent brute-force attacks.

Eliminate: if it’s not crucial to the business, remove the feature that causes the threat. For example, if our API allows users to supply URLs from which our system can pull data, such as images or company details, malicious actors could exploit this feature to run server-side request forgery (SSRF), port scanning, and other types of attacks. If this feature isn’t crucial for our application, we may want to remove it.

Transfer: if the cost of addressing the threat is too high, we may want to shift responsibility to another agent, such as the customer. For example, if our API allows creating shareable links for resources containing highly sensitive data, we may decide it’s the responsibility of the user to ensure the link only lands in the right hands and that they disable it when suspicious access is detected.

Accept: if the threat poses an acceptable level of risk for the business, don’t address it. For example, if our API gives discounts to customers who enroll other users on our platform, malicious actors can exploit this feature by registering fake users. However, if our website is in an early stage of growth, we may decide that this is a small price to pay to grow our user base.

Our threat responses have a direct impact on the business. Mitigate means putting in additional work to harden the system, while eliminate means removing features from our application, and transferring means changing the liability model in our business. For this reason, it’s important to bring the relevant stakeholders to the table to decide the most desirable response to each threat.

2.2.4 Review and validation

Once we’ve defined how we respond to our threats, it’s time to answer the final question: “did we do a good enough job?” This is a good time to review all our previous work and come together as a team. Do our DFDs represent our system accurately? Did we identify all the threats? Did we bring the relevant teams and stakeholders into our threat modeling and resolution exercises? Can we verify that our threat response is working? Run a retrospective with your team to reflect on your threat modeling processes and identify improvements for the future, and if possible, engage other teams to review your threat models and offer feedback from different perspectives.

DEFINITION

In agile software development, a retrospective is a team meeting that happens at the end of an iteration or project to discuss opportunities for improvement for future iterations, and identify things that worked well and the team should keep doing. To learn more about retrospectives, check out Aino Vonge Corry’s Retrospectives Antipatterns (Addison-Wesley, 2020).

An important question at this point is: when do we do threat modeling? The answer is all the time! Threat modeling should be a continuous exercise to assess your system vulnerabilities and bring all stakeholders together on the importance of API security. Our systems are bound to evolve and change all the time, so it’s a good idea to run threat modeling exercises every few months (e.g. every quarter or twice a year) to re-assess our system. If you’re going to make major changes to your applications, it’s also a good idea to kick off those changes with a threat modelling exercise. Ideally, we start doing threat modeling early in our API design process. This allows security to be “built into” your APIs rather than “bolted-on” to them[6].

2.3 Act now!

When and how do you start tackling API security? As you learn more about API security and how it affects your organization, you may want to tackle all of it at once. But here’s the reality: you don’t go from zero to hero overnight. The journey to becoming a mature organization in API security can take months or even years. If you start your API security journey by taking over the most complex problems, progress may feel slow and demoralizing for you and your team. My recommendation is to start by picking low-hanging fruits that have the biggest impact on your organization. This will help to build momentum and confidence in your ability to tackle API security.

What are those low-hanging fruits? It depends on where you are in your API security maturity journey. Go back to table 2.1 to determine your level of maturity. If your level of API security readiness is low, start there. For example, many organizations struggle to keep their API documentation up to date, and some organizations don’t even have proper documentation at all, which means they don’t know what their attack surface looks like. If that’s your case, start there (see section 3.5 for more details on how to document your APIs).

What if your attack surface is already under control? Keep on reading - in this section, I describe actionable steps any organization can take to improve their API security posture, including strengthening your authentication and authorization system, using proper and up-to-date API libraries, and leveraging cloud protection tools. Let’s break it down!

2.3.1 Document your APIs

According to Salt Security’s State of API Security Q1 2023, only 19% of organizations feel confident that their API documentation is accurate.[7] If you fall into the other 81%, that’s bad news for your organization and great news for your hackers. As you’ll learn in section 3.5, documentation is API security 101, and there are different forms of documentation. In this case, we’re looking for a formal API specification. Without a formal specification, you can’t validate that your API is correctly implemented, and you can’t run security tests against its design. From my experience working with organizations that sell products and services through their APIs, outdated and/or inaccurate API documentation also means unhappy clients and loss of business.

If you lack a formal API specification or don’t have a specification at all, I recommend you work on that first. As we explained in section 1.2.1, you can you various approaches to produce an API specification, including using tools that generate specifications from code and employing an API writer to produce the specification by looking at your code. If your API is consumed by a web application, you can use traffic interceptors like Burp to capture API traffic and infer the schemas from the payloads. If your APIs are too complex and none of those approaches work, engage a specialized service to help you in this area.

API specifications document the design of our APIs, and as you’ll learn in chapters 4, 5, and 6, we can tackle many API vulnerabilities by hardening the design. Once you have a formal specification in place, the next best step is to use it to run security tests against the specification and to validate that your API implementation is correct. In chapter 12, you’ll learn to run such tests.

2.3.2 Strengthen authentication and authorization

Another low-hanging fruit with great impact for your business is authorization and authentication. If you look at the OWASP top 10 API security threats (https://owasp.org/API-Security/editions/2023/en/0x11-t10/, see also chapter 5), you’ll see 4 of them relate to weak authentication and authorization (API1:2023, API2:2023, API3:2023, and API5:2023), and according to leading API expert Eric Newcomer, authorization remains the biggest challenge to API security.[8] Indeed many APIs are built with the assumption that all authenticated requests are legitimate, which means proper access controls are often lacking on sensitive endpoints.[9] For example, during my consulting work, I’ve found plenty of APIs that fail to enforce user-based access controls on operations like DELETE and PUT, which means any authenticated user can update or delete resources owned by other users. That’s the fast lane for a data breach.

DEFINITION

Authentication refers to the process of verifying the identity of a user through a challenge, such as a combination of username and password, and in many cases, an additional one-time code if the application uses multi-factor authentication (MFA). Authorization refers to the process of verifying that a user has access to the request data and operations in your API.

If you’ve never done it before, I recommend you review access controls across all your endpoints. Even better, write automated tests so that you run these checks repeatedly. You’ll learn more about this type of testing in chapter 12, but as you can see in figure 2. 7, you can get started really simple. For example, create two users A and B, and a bunch of resources linked to each of them. Can user A access resources that only user B should have access to? If that’s the case, you’ve got a Broken Object Level Authorization (BOLA) vulnerability. Make sure it’s fixed and run the same test on all future changes to the code. This is tedious work, and the bigger and more complex the application, the longer it’ll take to complete the test suite. Don’t despair though and proceed in small steps. Fixing one endpoint at a time is already a huge win for your organization.

Figure 2.7 Start testing your access controls with simple scenarios. For example, create two users A and B. Make a payment with user A and check if user B has access to the details of user A’s payment.

Another common vulnerability relates to weak authentication. Authentication and authorization are complex processes. When it comes to APIs, we have a ton of standards we must master to implement authentication and authorization securely (we’ll review them in chapters 7 and 8). Sadly, many organizations fall into the trap of believing that they can bypass all these standards and build their own, simpler authentication and authorization systems. The result is often weak authentication systems with vulnerabilities that threat actors can exploit to take over other user accounts or to obtain admin credentials.[10] If you’ve gone down the route of building your own authentication and authorization system, I recommend you audit the system as a matter of urgency. You’ll likely find a surprise or two.

What about your access tokens? The Gold Standard for APIs is the JSON Web Token (JWT) specification, which we’ll study in detail in chapters 7 and 8, together with its alternatives. JWTs are JSON documents that contain information about the right of a user to access our API, and they contain a signature that proves that they are legitimate. JWTs can be very powerful and secure, but it’s easy to misconfigure them and lose all their benefits. For example, do your JWTs contain personal or sensitive information? Because they shouldn’t. What signing algorithm do you use? Does your JWT validation library allow random algorithms to be specified? Because if that’s the case, your JWTs are highly vulnerable. And last but not least: are you sure you are using access tokens and not ID tokens?

2.3.3 Use proper API libraries

One recommendation I make to all organizations I work with is to use proper API libraries. APIs are deceptively simple. After all, how difficult can it be to handle URL query parameters and request payloads? As it happens, it’s really complicated, which is the reason we have entire libraries and ecosystems dedicated to this.

Your best chance of delivering a solid API is by using one of those libraries. For example, if you are building in Python and have a Flask web server, and you want to add an API, use a proper Flask API library like APIFlask (https://github.com/apiflask/apiflask) or Flask-smorest (https://github.com/marshmallow-code/flask-smorest). If it is Django, use the Django Rest Framework (https://github.com/encode/django-rest-framework) or Django Ninja (https://github.com/vitalik/django-ninja). Good API libraries have built-in support for input and output validation, data serialization, error handling, and excellent support for the semantics of the API style you wish to build, like OpenAPI for REST APIs and the Schema Definition Language (SDL) for GraphQL.

NOTE

Hexagonal architecture (HA), or architecture of ports and adapters, is a widely used pattern for decoupling an application’s business layer from its external dependencies, such as databases, third-party integrations, and its interfaces. To learn more about HA, check out chapter 7 from Jose Haro Peralta’s Microservice APIs (Manning, 2022).

If you already have an API implemented without using proper API libraries, you’ll have to migrate. To facilitate migrations from one library or framework to another, I recommend using hexagonal architecture (HA). HA, or architecture of ports and adapters, decouples your business layer from external dependencies (ports), and it uses “adapters” to connect to them. The API layer of your application is where you declare the routes and process the requests, and in HA, that’s a port. Since the API layer is decoupled, moving to another framework simply requires updating your API layer. As always, do one step at a time. Start with a small portion of your API and migrate your endpoints one at a time. You’ll feel the benefits immediately.

2.3.4 Leverage cloud protection tools

Finally, if you don’t currently use an API Gateway or a Web Application Firewall (WAF), using those technologies will immediately improve your API security posture. API Gateways can handle secure access token validation, DDoS protection, rate limiting, data validation, and more. In many cases, API Gateways also allow you to minimize your surface attack area when you configure them as the single-entry point to your platform. For example, many organizations struggle with shadow APIs, which are undocumented APIs and endpoints in your code.[11] API Gateways allow you to declare which endpoints can be exposed, which minimizes the risk of exposing shadow endpoints. Similarly, WAFs are firewalls that inspect HTTP traffic and protect applications from well-known web threats, such as SQL injection and server-side request forgery (SSRF). Modern WAFs can be fed with API specification to tailor their protection to the specific needs of our APIs.

You can combine an API Gateway with a WAF for further protection. WAFs provide an additional layer of protection against suspicious requests, SQL injection attacks, bots, and more. Just one word of warning here: research shows that technologies like API Gateways and WAFs give organizations overconfidence in their API security posture.[12] Don’t fall into that trap. API Gateways and WAFs are useful but insufficient to fully protect your APIs. For example, neither of them will protect you against OWASP’s API1:2023 Broken Object Level Authorization, which happens when users can access resources that don’t belong to them.

As you can see, there are plenty of things you can do right now to start tackling API security at your organization. These are the low-hanging fruits that will deliver the most value. As you mature in your API security journey, the obvious question is, how can you formalize your approach to security and ensure everybody is aligned with it? That’s the topic of the following sections.

2.4 Creating an API security program

As your organization grows and your APIs get bigger and more complex, the obvious question is, how do you ensure your APIs remain secure? Many API security initiatives begin with a single advocate. This could be you, dear reader. As you grow aware of the importance of API security, you may want to champion API security within your organization. What you will realize is that API security is not the job of one person or even one team. API security is everybody’s job: security must be part of the API development process. How do you do that? With an API security program.

An API security program is an outline of your API security strategy. It describes your vision and goals, and how you plan to achieve them. If you’ve never tackled API security at the organizational level within your company, don’t be too ambitious at the beginning. You need a realistic view of what you can accomplish, and you need to get buy-in from your organization (see section 2.5). Crucially, you need Governance (yes, with capital G) to ensure all this comes together and your API security strategy gets executed.

Leading API expert Lorna Mitchell lays out the ingredients of successful API Governance, or “Governance without tears,”[13] as she puts it:

Write down your standards and keep them lightweight. This documentation is crucial to keep everybody on the same page with regards to API security, so it should be succinct and to the point, and easily accessible (e.g. in a GitHub repository as opposed to a random page on Confluence). You can make it even more helpful if you include code examples and references to external sources.

Get lots of input from other stakeholders within your organization, including architects, developers, product managers, and so on. People are more likely to contribute when they feel listened to, so this is a fantastic way to get buy-in.

Automate. If you’ve been in software for a while, you know it’s very difficult to enforce rules unless their enforcement is automated. For example, leverage security-aware linters like Spectral to test your API designs and fuzz testers like Schemathesis to validate your API implementation.

TIP

Bounty programs are an excellent way to outsource the discovery of your application’s vulnerabilities, allowing you to tap into talented researchers all over the world. Bug hunters disclose their findings responsibly in exchange for a compensation payment. You can run your own bounty programs, or you can use platforms like HackerOne (https://www.hackerone.com/) to manage your bounty program. Before starting a bounty program, make sure you have resolved all vulnerabilities known to you and that you’ll have the resources and capacity to act on bug hunter feedback. Also, make sure your program is transparent and compensates bug hunters adequately for their findings.

As your organization grows more mature and confident in API security, you may want to take things to the next level. If you get sufficient support from your organization, this is the time to go beyond the low-hanging fruits and address more complex issues. For example, include API security training as part of your program, draft your incident response plans, or work on advanced threat detection techniques (we’ll discuss some in chapter 11).

DEFINITION

Many organizations create so-called red, blue, and purple teams to improve their cybersecurity posture. But what do those teams do? Red teams simulate threat actor behavior, attempting to run attacks and exploit vulnerabilities in your application. Blue teams focus on threat detection, prevention, and response –their job is to identify and mitigate the attacks from the red team. Purple teams analyze the performance of red and blue teams to discover opportunities for improvement.

You can also run internal threat hunting sessions within your organization, set up red, blue, and purple teams, and if you feel strong about your security posture, start a bounty program. To do all this, you’ll need buy-in from the rest of your organization, which is the topic of the next section.

2.5 Aligning API security with your organization

Most organizations want to deliver software as fast as possible, often at the cost of security. However, you can only push this approach so far before you get a data breach. As Corey Ball nicely puts it, “if you don’t test it [your API], then a cybercriminal somewhere is going to do it for you.”[14] In my experience, most organizations take API security seriously, but they often lack knowledge or guidance on how to secure their APIs effectively. A common solution is to delegate API security to cybersecurity teams. You do need someone at your organization to champion security, someone to hold accountable for the success of your cybersecurity program. Ideally, a chief security officer (CISO) or a role with similar responsibilities. However, one person or one team alone won’t get you far. As we saw in previous sections, API security is not a one-person or one-team job. We need everybody’s involvement.

The reason why API security is everybody’s job is that everything in our API, from the design of user flows to the choice of query parameters and the implementation details has a direct impact on our API security posture. Furthermore, as we saw in chapter 1, traditional application security isn’t sufficient to protect our APIs. We need a more comprehensive approach that covers design, implementation, architecture, and operations.

Addressing API security properly may appear to add more work to your organization and hence slow down the speed of software delivery. Suddenly, your product managers must take security into account when designing user flows, developers must get familiar with proper API implementation libraries, learn to work with JWTs correctly, and study standard protocols such as Open Authorization (OAuth) and OpenID Connect (OIDC). They must run robust and comprehensive tests against the API design and implementation. How can you justify this to your organization?

Consider the cost of not doing it. If you fail to tackle API security properly, you risk having a data breach. Data breaches often come with hefty penalties and result in loss of trust towards your business. According to research by IBM, the global average cost of a data breach in 2023 was $4.45 million.[15] For small organizations, a data breach could mean the end it. And unfortunately, API data breaches are on the rise often due to very basic mistakes. For example, in September 2022, Optus, Australia’s second-largest telecommunications company, suffered an API data breach that affected up to 10 million customers. The breach included personal details such as names, home addresses, phone numbers, and even passport and driving license numbers.[16] The cost? A whopping $140 million to audit the breach and compensate the affected customers.[17]

It should be clear that you cannot afford not to take API security very seriously. The question is, can you really afford proper API security? Will your software delivery pace slow down, and will your organization lose a competitive edge consequently? Not really, if you do it well that is. Figure 2.8 shows the typical API development cycle in many organizations. As you can see, we go through the following stages:

We start with an API design stage,

then proceed with implementation,

after that we deploy our changes, and finally,

we run a battery of security tests, typically manually.

If we find a security vulnerability, we must go back to the beginning of the process, redesign the API, reimplement, re-deploy, and security test again. Rinse and repeat after no issues are found. Not all organizations begin their API development process with a design stage, hence why it is marked with “not always” in figure 2.8.

Not only is this process slow, but as Dan Barahona highlights, it doesn’t scale,[18] and in the interest of speed, organizations end up cutting corners in their security testing strategy.

Figure 2.8 The typical API development cycle goes through a design stage, then implementation, and then deployment. Once deployed, we run functional and finally security tests. If we find any issues during the test stages, we go back to the beginning of the process, redesign, reimplement, re-deploy, and test again.

API security by design takes a different approach to API security. As illustrated in figure 2.9, with security by design we address security from the very beginning of the API development process – from the design stage. It all begins with the design of safe user flows and safe input parameters. We test our API design for vulnerabilities and only proceed to implementation when the design is secure, or when the risk profile of our design is deemed acceptable by our threat modeling. Then we build the API following secure implementation patterns and we validate that the implementation is correct. By the time we deploy and run our security tests, we’ll find a lot fewer issues to fix and hence will need fewer repetitions of the cycle shown in figure 2.8. This means we can release our APIs faster and with more confidence in our security posture.

Figure 2.9 With API security by design, we address security from the API design stage and through every stage of development.

Make sure you can prove to the business that your new approach to API security is working. How do you do that? Use metrics. As API operations (APIOps) expert Ikenna Nwaiwu highlights, measurable outcomes are one of the most effective communication tools, and they help you gain buy-in from your organization.[19] For example, how many times do you currently go through the cycle on figure 2.8? How long does it take you to move an API from design to deployment in production? Do you have an API specification? How many security errors does it have when you test it with Spectral’s OWASP plugin (you’ll learn about this in chapter 12)? How many errors does a fuzz tester like Schemathesis (again, chapter 12) show now when you run it against your API? Keeping track of these and similar metrics will help you make the case for a robust approach to API security, and your organization will thank you later.

2.6 Navigating API security audits

In an ideal world, organizations start improving their API security posture on their own initiative. In many cases, though, it takes an audit to make the wake-up call. Audits can happen for many reasons. For example, investors generally audit your system before they put money into your business. If your business is going to be acquired, the buyer generally wants to know they are buying a well-architected platform. If you enter a partnership with a new business, the other business will want to know that your APIs are safe and secure. Often, instead of running a customer security audit on your platform, investors, acquirers, and business partners will require you to have a security certification, such as ISO/27001, HIPAA, SOC, or PCI DSS. This is particularly true in sensitive sectors such as healthcare, finance, and defense.

How do you handle these audits from an API security perspective? The first step is to perform a gap assessment, which you can do with the help of the guide in section 2.1 of this chapter. My recommendation is to be ready to answer detailed questions about your security posture. Go back to table 2.1 and make sure you get a handle on most of the questions in the table. For example, there’s a world of difference between not knowing what your API attack surface looks like at all and having a provisional map; even better if you can provide a percentage of the mapped surface. You must show commitment to continuously improve your security posture, and sometimes stakeholders like investors will demand regular updates.

The goal of a gap assessment is to understand how far you are from successfully passing an audit. The nature of your data is critical in this exercise. If you work with highly sensitive data, like healthcare and financial data, you’ll face more scrutiny from the auditors, and therefore, you’ll want to be able to answer successfully most of the answers in table 2.1 before you engage the auditor. Once you understand the areas where the gap is wider, you can set up steps to fill those gaps and ensure you pass the assessment with flying colors.

You will never get as much support from your organization to improve your API security posture as in the face of an audit. You won’t be able to tackle it all at once, but this is a good opportunity to map out your API security strategy if you don’t have one yet. It’s also an excellent opportunity to start working on some of the low-hanging fruits we discussed in section 2.3. If this is your first audit, nobody expects you to be perfect. What is more important is to show awareness of your security technical debt, a roadmap for addressing it, and a commitment to doing it. Don’t forget to define some measurable outcomes as we discussed in section 2.5. Over time, you’ll be expected to show progress and more maturity, and metrics are one of the best ways of doing it.

Finally, remember there’s always professional help around if you need it. API security audits represent are one of the most successful business models in recent years, and many companies offer services in this space, including leading organizations in this field like 42Crunch, APISec, and SWO2, and others. Just bear in mind that using external service providers is not an excuse for not creating your own API security strategy. If you have APIs, you must own your API security posture management and work towards an API security by design methodology.

2.7 Summary

Your API security journey begins with an evaluation of your API security posture. Organizations mature in API security have:

Data inventory, i.e. they know what sensitive data they expose and where.

Attack surface map, i.e. they know all the endpoints available on their APIs.

Authentication and authorization following best practices and standards like OAuth, OIDC, and JWTs.

Robust access controls ensure users can’t get their hands on the wrong data.

Secure implementation ensuring strict compliance with the API specification.

Automated security testing using design-testing tools, fuzzy testers, and custom test suites.

Risk analysis allows us to prioritize known vulnerabilities quickly.

Incident detection that gives real-time visibility over suspicious user behavior and security incidents.

Incident response plans mean we know what to do when an incident happens.

Continuous training to raise awareness about API security vulnerabilities.

Threat modeling is a great exercise to get a holistic view of your API vulnerabilities and foster team collaboration.

Good threat models answer the following questions from the Threat Modeling Manifesto:

What are we working on?

What can go wrong?

What are we doing to do about it?

Did we do a good enough job?

To address the Threat Modeling Manifesto’s questions, we break down threat modelling into the following steps:

Application decomposition

Threat identification and ranking

Response and mitigations

Review and validation

We use methodologies like STRIDE to model threats. STRIDE identifies the following threat categories:

Spoofing: risk of stealing user credentials.

Tampering: risk of performing unintended updates on our data.

Repudiation: risk of not being able to detect malicious activity.

Information Disclosure: risk of leaking sensitive data.

Denial of Service: risk of system unavailability.

Elevation of Privileges: risk of an attacker assuming admin privileges.

The best way to start tackling API security is by picking low-hanging fruits like mapping out your attack surface, fixing your API documentation, or adopting authentication and authorization standards.

An API security program is an outline of your API security strategy and how you implement it.

You need buy-in from your organization to push forward your API security program:

Get feedback from your stakeholders to encourage their involvement.

Create realistic metrics that show how your organization benefits from this.

To prepare for an API security audit, ensure you can answer detailed questions about your security posture. You must show awareness of current technical debt and commitment to address it.

[1] For a summary of the main cybersecurity frameworks, see Kim Crawley, 8 Steps to Better Security (Wiley, 2021), pp. 89-108.

[2] Corey Ball, “Earning Confidence in your API Security”, presented at APIDays Paris, December 6-8, 2023.

[3] Repudiability in the original paper.

[4] Elevation of Privilege is a threat modeling card game created by Adam Shostack in 2010 (https://shostack.org/games/elevation-of-privilege).

[5] There are various open-source tools to test the strength (or break) of JWT signatures. You can read about them on OWASP’s website: https://owasp.org/www-project-web-security-testing-guide/latest/4-Web_Application_Security_Testing/06-Session_Management_Testing/10-Testing_JSON_Web_Tokens.

[6] Pun borrowed from OWASP’s Threat Modeling Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html).

[7] Salt Security, State of API Security Q1 2023, p. 12 (https://content.salt.security/rs/352-UXR-417/images/SaltSecurity-Report-State_of_API_Security.pdf). Check out also Radware’s The 2022 State of API Security, which found that 62% of organizations have over 30% of their APIs undocumented.

[8] Eric Newcomer, “API Security: Is Authorization the Biggest Threat?”, The NewStack (June 12, 2023, https://thenewstack.io/api-security-is-authorization-the-biggest-threat/).

[9] According to Salt Security, 78% of API attacks are performed by authenticated users (State of API Security Q1 2023, p. 4).

[10] According to Salt Security, 40% of API security incidents in Q1 2023 were related to authentication, and account takeover is the second most concerning vulnerability for organizations (State of API Security Q1 2023, pp. 4 and 8).

[11] We’ll discuss shadow APIs at length in section 3.4, but for a quick overview, check out Nick Rago, “Are You Haunted by Zombie, Shadow and Ghost APIs?”, Salt Security Blog (Oct 28, 2022, https://salt.security/blog/are-you-haunted-by-zombie-shadow-and-ghost-apis).

[12] According to Radware’s The 2022 State of API Security, “there is a false belief in the adequacy of API gateways and traditional WAFs providing adequate protection of APIs against both vulnerability and automated bot exploits,” with 28.6% of organizations using API Gateways and 21.2% of organizations using WAFs as their primary method for identifying API attacks.

[13] Lorna Mitchell, “API governance without tears”, talk presented at APIDays London, September 19, 2023 (slides available at https://noti.st/lornajane/PiwFuR#sxdiYSF).

[14] Quote taken from “API Security 101: Establishing & Managing a Secure API Program,” webinar by Andrew Binder and Dan Barahona, hosted by Redspin on February 11, 2021 (recording available on YouTube: https://youtu.be/cMN6yr4nBUo; the quote appears around minute 36:20).

[15] IBM, Cost of a Data Breach Report 2023.

[16] Andrew Greene, “Optus rejects insider claims of 'human error' as possible factor in hack affecting millions of Australians,” ABC News (September 23, 2022, https://www.abc.net.au/news/2022-09-23/optus-rejects-claim-hack-likely-result-of-human-error/101468846). Check out also the analysis by Traceable within the context of the Telecom industry: “The Telecom Industry: Why APIs Are Becoming their Worst Nightmare,” Traceable Blog (https://www.traceable.ai/blog-post/the-telecom-industry-api-security-worst-nightmare).

[17] Zoe Samios, “Optus hack to cost at least $140 million,” The Sydney Morning Herald (November 10, 2022, https://www.smh.com.au/business/companies/optus-puts-aside-140m-to-replace-customers-hacked-identity-documents-20221110-p5bx4g.html).

[18] Dan Barahona and Andrew Binder, “API Security 101: Establishing & Managing a Secure API Program,” webinar by Andrew Binder and Dan Barahona, hosted by Redspin on February 11, 2021 (recording available on YouTube: https://youtu.be/cMN6yr4nBUo).

[19] Check out Ikenna Nwaiwu’s talk “From 2.5 weeks to 1 day: Improving API delivery outcome”, talk at APIDays London, September 19, 2023 (recording available on YouTube: https://youtu.be/4GLD0DTI8DI), and his upcoming book Automating API Delivery (Manning, 2024), in particular chapter 11.

3 API security principles

This chapter covers

What shift left means for API security and its impact on the development cycle

The Zero Trust Security model and how it applies to APIs

Why we must secure our internal APIs

The importance of API documentation for security by design

Where, how, and why to validate data in our APIs

The role of security in continuous delivery

You’re building an API, and halfway through the implementation, you start wondering whether the API will be secure. As you approach the release date, your manager also wants to confirm with you that security is being taken care of. Scrambling for an answer, you put together an API security checklist by looking for online resources such as Shieldify’s popular API-Security-Checklist (https://github.com/shieldfy/API-Security-Checklist). You go through the whole checklist and tick all the boxes. You’re confident you’ve addressed security in your API. As you approach the date of the release, your QA and cybersecurity teams run a battery of tests against the API. The results come positive. All seems to be good. You move forward with the release. Two weeks later, you have an API breach. How could it happen? The problem with this approach is it leaves security for the last minute. It treats security as a second-class concern.

So, how do we make our APIs secure? As you’ll learn in this chapter, security begins with our API design. Are our user flows safe? Do we have clearly differentiated read-only versus write data models? Are we exposing server-side data in user input? Is user input sufficiently constrained? And more importantly: is our design accurately documented? One of the fundamental principles in API security is you can’t protect what you don’t know, and in this chapter, you’ll learn to leverage API documentation for security.

Security also plays a role in our technology and implementation strategies. For example, are we using proper API and data validation libraries and frameworks? A very common mistake is to start creating APIs without those frameworks, which often results in poorly built APIs with big security holes. As you’ll see in this chapter, we usually have libraries for generic tasks like building APIs, handling data validation, and validating access tokens, and we stand our best chances of delivering secure APIs by tapping into those libraries.

Over the course of this book, you will learn about the most common API threats and design vulnerabilities, and how to prevent them by leveraging design, implementation, and architectural patterns. The basis of those patterns is a collection of security principles like shift-left security and the zero-trust security model. In this chapter, you learn about these security principles and how they help you improve your API security posture. We’ll begin with the most important principle: shift left security.

3.1 Shift left API security

There are two fundamental approaches to software security: we can retrofit security onto our applications, or we can build security into them. As shown in figure 3.1, the problem with retrofitted or bolt-on security is it treats our software as a black box and checks for generic vulnerabilities like lack of SSL, weak encryption, cross-site scripting (XSS), configuration leak, unprotected endpoints, and so on. As Dan Barahona, founder of APISec, highlights, this is not sufficient for API security, as most API breaches are due to flaws in business and access control logic.[1]

Figure 3.1 Traditionally, we run API security tests at the end of the software development life cycle (SDLC). This approach is also known as retrofitting security onto our applications.

For example, in 2019, PingSafe found a vulnerability in Uber’s API that allowed an attacker to take over other user accounts as a result of a series of vulnerable user flows. By supplying a valid phone number on an endpoint, PingSafe was able to obtain the corresponding user ID if a match was found. This information could then be supplied to the getConsentScreenDetails operation, which leaked the user’s access token.[2] Uber fixed this vulnerability by hardening the API access controls and redesigning the responses to expose less-sensitive data.

Figure 3.2 To build secure by design APIs, we must address security early in the SDLC and shorten the feedback loop by assessing our vulnerabilities frequently. This is known as building security into our applications.

Thankfully for Uber, this vulnerability was discovered as part of their bug bounty program. Many attackers, however, won’t be gracious when they discover vulnerabilities in your APIs and will rather exploit them to their benefit. To prevent those vulnerabilities, we must build security into our APIs, and this means addressing security early in the development cycle. This is what we call API security by design.

definition

A bug bounty program is a program through which organizations offer compensation to individuals who report bugs, security exploits, and other vulnerabilities on their websites. When run properly, bug bounty programs are very effective to uncover critical vulnerabilities on our websites. A popular platform for running bug bounty programs is HackerOne (https://www.hackerone.com/).

As shown in figure 3.2, everything starts in the design stage. Are our user flows vulnerable? Are we exposing too much sensitive data in certain operations? Do we have robust access controls? How do we ensure users don’t override server-side properties? Which endpoints and parameters must be flexible by design and hence require extra security checks? Addressing these types of questions at the beginning of our API journey is the best way to build security into our APIs.

DEFINITION

Security by design is a paradigm in cybersecurity that applies security into every step of the software development life cycle (SDLC). The idea is to address vulnerabilities at every stage of the SDLC before we move on to the next to build on top of foundationally secure software.

Tackling security during the design stage is a golden opportunity to create a solid base for our APIs. But API security by design doesn’t stop there – the goal is to approach every step in the API development process with a security-by-design perspective. For example, do we use secure implementation patterns? Does the implementation follow the API specification accurately? Do we have automated tests to validate access controls? By injecting security into every step of the development process, we’re less likely to find vulnerabilities at the end. As we saw in chapter 2, this allows us to release more quickly and with more confidence.

DEFINITION

An API specification is a formal description of your API using a documentation standard interface description language (IDL) such as OpenAPI for REST APIs and the Schema Definition Language (SDL) for GraphQL APIs. For more on the benefits of using IDLs, see section 1.4 of Jose Haro Peralta’s Microservice APIs (Manning, 2022).

API security by design is an example of what we call shift-left on security. Shift-left is a paradigm in software development that encourages us to tackle issues early by building quality into our systems. The benefits of shifting left on security have been well-established for many years, as evidenced in the research conducted by the team behind the DevOps Research and Assessment (DORA).

Note

The DevOps Research and Assessment (DORA) team was founded by Nicole Forsgren, Jez Humble, and Gene Kim to study the methodologies that allow development to deliver high quality software fast. DORA publishes a yearly report titled State of DevOps and is now part of Google Cloud. To learn more bout DORA, see Jez Humble’s article “DORA’s Journey: An Exploration”, Medium (Feb 2, 2019, https://medium.com/@jezhumble/doras-journey-an-exploration-4c6bfc41e667).

Back in their 2015 State of DevOps Report, DORA found that organizations that tackle quality issues early were more likely to release more often and with more confidence (https://services.google.com/fh/files/misc/state-of-devops-2015.pdf). The same applies to security. The 2016 State of DevOps Report found that organizations that build “security into their daily work, as opposed to retrofitting security at the end… spent significantly less time addressing security issues.” (2016 State of DevOps Report, p. 28 [https://dora.dev/publications/pdf/state-of-devops-2016.pdf]). Later DORA reports confirm the same findings (the full list of reports is available here: https://dora.dev/research/).

DEFINITION

Shift-left is a paradigm in software development that builds quality into the software by encouraging frequent testing and delivering features in small batches. This philosophy borrows from W. Edwards Deming’s concept of Total Quality Management (TQM). In his Fourteen Points for the Transformation of Management, Deming encourages managers to “Cease dependence on inspection to achieve quality… by building quality into the product in the first place.” (W. Edwards Deming, Out of the Crisis, The MIT Press [Cambridge, MA), p. 22).

APIs are no exception. According to S&P Global’s 2022 API Security Report, 35% of respondents reported projects being delayed due to API security concerns. Of those, 87% believe the delays could have been prevented by incorporating security into the daily workflow.[3] And it’s not just the speed – shifting left on API security ensures we are less likely to run into security incidents in production.

To illustrate the benefits of shifting left on API security, imagine we have an e-commerce application with an API whose GET /products endpoint allows customers to browse the product catalog. Suppose we want to add a sort_by parameter to a GET /products endpoint so that customers can sort products by factors such as price, average review, and so on. Two common types of vulnerabilities in this type of parameter are SQL injection and property enumeration. Property enumeration is an exploit that allows threat actors to discover fields of our models that are not supposed to be exposed to external users. In an e-commerce site, one such field could be stock, which indicates the number of items per product left in stock. As illustrated in figure 3.3, a threat actor could use the sort_by parameter to sort items by stock available. Such information would be useful in scalping, which as we saw in section 1.3, is the practice of buying out the whole stock of a product to resell it at a higher price.

Figure 3.3 A threat actor discovers a server-side property called stock, which allows them to sort products by their available stock. Later, they may use this information to run a scalping attack on scarce products.

With the bolt-on security approach, we won’t detect any vulnerabilities around the sort_by parameter until we deploy the API to a test environment and run the security test suite. Once we run the security test suite, we may find out that the parameter is vulnerable to SQL injection, but we are unlikely to discover that it is vulnerable to property enumeration too. Then a likely fix is applying a regular expression on the parameter to prevent SQL injection, for example by restricting the value to a single word without spaces. However, as indicated in figure 3.4, this approach still leaves a hole in the API. By allowing users to sort items by any word, they can still run property enumeration attacks and gain knowledge about the hidden properties of the product model.

Figure 3.4 Bolt-on security is more likely to lead to a patched approach to security. For example, if our tests uncover a SQL injection vulnerability, we patch the API with a regex pattern, leaving the API still vulnerable to property enumeration.

Security by design addresses this problem from the start. For example, as highlighted in figure 3.5, we constrain sort_by to an enumeration with values price and review. Thanks to this constraint, we remove both the SQL injection and the property enumeration vulnerabilities by design. Now that we have a robust and secure design, the next task is ensuring that the implementation applies to the sort_by parameter the constraints we included in the specification. A great way to validate the implementation against the specification is using an API fuzzer, as you’ll learn in chapter 12.

Figure 3.5 Built-in security addresses the problem by design, restricting values using an enumeration based on our data model. By doing this, the API becomes resilient to SQL injection or property enumeration attacks.

A word of warning. Shifting left on security brings many benefits, but it’s not a silver bullet and it should not be your only security strategy. First, for shift-left security to deliver value, you must meet the following criteria:

Determine sensitive user flows at design time and ensure they are adequately protected. Work together with security experts within your organization to design secure payloads, minimize data exposure, constrain user input, and so on. Write test cases to ensure the implementation meets these requirements.

Document the API accurately: as we’ll see in section 3.4, API documentation plays a crucial role in security since it allows you to catch vulnerabilities at design time and more, and in chapter 12, you’ll learn to assess the strength of your design using the API specification.

Use the right tooling to test your APIs at design and run time. The API security vendor ecosystem is full of tools that run very generic checks against your APIs. Audit your tools before you use them and ensure they meet your testing requirements (you’ll learn more about this in chapter 12).

Shift-left security is also not a substitute for a robust battery of QA and cybersecurity tests at the end of your Software Development Life Cycle (SDLC), and those tests will often capture new vulnerabilities. And remember, security incidents are not a matter of if, but when. Shift-left security works best in combination with robust observability, automatic threat detection and response, and runbooks for what to do when a breach happens. The rest of this chapter discusses the principles you need to apply for a proper shift-left security strategy, beginning with a crucial question: what can we trust?

3.2 Zero Trust APIs

Modern web applications consist of complex data flows. As represented in figure 3.6, a single operation may require processing data from a user, pulling additional data from a third-party API, running checks with internal services, and publishing events to a messaging queue for further processing. The question every security architect must tackle here is: what can we trust in these flows? Can we trust user input? Can we trust third-party APIs? Can we trust internal services? What about our own databases?

Figure 3.6 Modern applications consist of complex data flows, like the request flow when a client sends a request to our API server, or the response flow when we reply to the request, and the third-party API flow when we connect with external services.

The best answer to these questions is the Zero Trust Security Model. The concept of Zero Trust Security was introduced by John Kindervag while at Forrester Research, a leading cybersecurity research organization.[4] Kindervag’s model identifies trust as a core vulnerability in our systems, and it recommends denying access by default to all traffic regardless of origin. This is often summarized as “don’t trust anything, always validate.” Kindervag also recommends actively monitoring, inspecting, and analyzing all traffic to identify sources of suspicious activity.

Figure 3.7 NIST 800-207’s Zero Trust architecture model enforces robust access controls and active monitoring of our traffic to detect malicious behavior.

In recent years, the concept of Zero Trust Security has matured, and in 2020, the National Institute of Standards and Technology (NIST) developed a framework of best practices for implementing Zero Trust architecture known as NIST 800-207.[5] As illustrated in figure 3.7, NIST 800-207 defines the following principles for Zero Trust Security:

We treat all data sources as resources.

We validate all traffic regardless of origin.

We enforce user-based access controls following a least-privileged model, namely, users can only access resources they own or are otherwise authorized to inspect and/or manipulate.

We use dynamic policies to determine user access, namely, we can change user access permissions dynamically anytime.

We monitor the integrity of all our resources. If a resource is compromised, we restrict its access to protect our users and our system.

We check that all traffic has the expected authentication and authorization credentials.

We actively monitor all traffic to detect suspicious activity and evaluate the security posture of our system.

Figure 3.8 Zero Trust APIs apply NIST 800-207’s principles to protect all assets and endpoints, apply robust access controls, and validate data across all flows.

What does this all mean for APIs? It means removing trust from every corner of our system. Don’t trust any user, regardless of whether they are authenticated or not, and regardless of their role - even if they claim to have administrator privileges, and regardless of the API they’re requesting access to, including, or specially, internal APIs (see section 3.4 for more on internal APIs). What do we mean when we say we can’t trust any user? As shown in figure 3.8, the Zero Trust Security model has wide-ranging implications for API security. Zero Trust APIs protect all their endpoints, apply robust validation across all data flows, including requests, responses, third-party integrations, and more, and they actively monitor all activity to detect suspicious behavior. The below list enumerates the characteristics of Zero Trust APIs:

Protect all your endpoints.

Don’t expose unnecessary endpoints or operations.

Validate that every API call has the right attributes to access your system, e.g. it has the right headers, tokens, credentials, and so on.

Enforce user-based and role-based access controls to all resources following a least-privileged access model.

Validate data in all API requests, including payloads and URL query and path parameters.

Validate all data before sending it back to the users.

Validate all data from third-party APIs before processing it.

Validate all data from internal services.

Actively monitor user activity to detect suspicious behavior and react to it.

The first set of characteristics (1-4) relate to access controls. Zero Trust APIs protect all their endpoints. Most APIs have a concept of private and public resources. Public resources are accessible to everybody, while private resources are accessible only to their legitimate owners or users who were granted access to them. Even when an endpoint is publicly available to all users, we want to ensure it doesn’t get abused. For example, a product catalog may be accessible to all users on an e-commerce store, but we don’t want malicious actors to scrape all the content.

What about private or user-restricted resources? Most APIs authorize access based on JSON Web Tokens (JWTs). The JWT contains claims about the right of a user to access our API, their role and privileges, and an opaque ID that identifies the user, among other data. As represented in figure 3.9, our job is to check each claim to evaluate the right of a user to gain access to the requested resource.

Figure 3.9 To authorize a request, we validate all components and properties of the JWT, including the signature, the headers, and all the claims in the payload against their expected values.

Authorization is one of the biggest sources of vulnerabilities in APIs, especially when it comes to access controls.[6] For example, a common API security flaw is assuming all authenticated users are legit and hence relaxing access controls on authenticated operations. In fact, according to Salt Security, 78% of API attacks come from authenticated users.[7] This is critical in endpoints that expose sensitive data and operations, like updating or deleting resources. For example, in February 2022, a security researcher who goes by the name of Tree of Alpha discovered a flaw in Coinbase’s API that allowed users to sell assets they didn’t own.[8]

To prevent authorization breaches, apply the Zero Trust Security model from the very beginning of your API journey. When you’re designing your API, that’s a good time to decide which endpoints or operations need extra care from a security perspective. For example, as indicated in figure 3.10, on an e-commerce application, everybody may be able to browse the online catalog, but only authenticated users can upload new products, and only the owners of those product listings can update or delete them.

Figure 3.10 On an e-commerce platform, an unauthenticated user can browse the catalog but can’t upload new products or change their details.

Drawing a security profile of our API like in figure 3.10 gives us good visibility of where we must double down on access controls. We can leverage this information to write access control tests and verify that the API implementation complies with our security model. This is an excellent way to shift left our API security strategy. In addition to robust access controls, Zero Trust APIs must apply strict validation across all data flows, and the next section explains how we do that.

3.3 Validate everything!

There’s a popular saying in software that goes “never trust the client.” The idea is that we can’t control how users engage with our applications, so we must apply robust validation and sanitization to all data sent to our servers. Input validation checks that the data send to our API conforms to a given schema, while sanitization removes malicious elements from the data. For example, an e-commerce website that allows users to write product reviews may constrain the text to one thousand characters. Input validation checks that the text contains one thousand characters or less, while input sanitization ensures the input doesn’t contain malicious code or unexpected characters that can cause errors, harm our system, or cause unexpected behavior when processed. Let’s see some examples of this!

Figure 3.11 represents a common pattern for building modern web applications with a single page application (SPA) framework in the frontend (aka the client) and an API in the backend. The question often that arises in this context is: can we trust the client to send legit data to our servers?

Figure 3.11 A popular pattern for building web applications is to use single page application (SPA) frameworks like React in the frontend and connect them to an API in the backend.

As illustrated in figure 3.12, client applications help constrain user input and ensure users only send legit data, but nothing prevents them from going directly to our servers. In fact, modern applications often expose APIs in the backend, which users can legitimately use with any type of client, such as command-line terminals. What does this mean for our threat model? It means we can’t expect all users to send valid data, and threat actors will send malicious requests at the first chance.

Figure 3.12 Client applications ship with data validation functionality that helps ensure users only send valid data to our servers.

So, can we trust our users to send legit data to our servers? Absolutely not. When receiving data into our system, we must handle two types of risk:

Risk of malformed or invalid data

Risk of malicious data

Malicious data speaks for itself. It includes things like malicious SQL statements or execution commands that can cause havoc if they make their way into our system. What about malformed or invalid data? Malformed or invalid data is data that doesn’t conform with the assumptions of our data models. When malformed data makes its way into our system, it causes confusing errors and integration problems. For example, as shown in figure 3.13, our system may assume certain value constraints for some data models, and if we fail to enforce those constraints, our application will fail when the data is handled in other contexts.

Figure 3.13 The underlying data model in a book catalog API assumes book formats can only be “printed” or “eBook.” A user lists a new book setting the format to “digital.” When the data is loaded in a different context, it causes an error because the application can’t handle the invalid value.

To prevent the risk of malformed or malicious data, we must validate all data coming to our servers regardless of its origin. How about other data flows? As we saw in the previous section (figure 3.6, reproduced as 3.14 for convenience), modern applications consist of complex data flows, such as the request flow, the response flow, the database to application flow, and so on. Can we trust that the data flowing through those channels is safe and valid? No, we can’t. To understand why, let’s dive into some of these flows.

Figure 3.14 Modern applications consist of complex data flows such as the request flow, the response flow, the third-party API flow, and more.

As highlighted in figure 3.15, the third-party flow involves pulling data from a third-party API. In modern applications, this is a very typical data flow. We use third-party applications to outsource common yet complex tasks like sending emails, mapping coordinates, managing calendars, making payments, signing documents, pulling user profile data from third-party applications, and so on.

Figure 3.15 Modern applications use third-party APIs to outsource common yet complex tasks like processing payments, sending emails, or resolving a location’s coordinates.

What could go wrong here? By pulling data from third-party applications, performing operations with them, and storing data in their servers, we are directly exposed to their threat models. As indicated in figure 3.16, if a third-party application exposes a vulnerability, malicious actors may be able to inject malicious code or bad data into their systems, and the malicious data/code can make its way into our systems. This is known as Unsafe Consumption of APIs and is considered one of the top vulnerabilities in modern APIs.

TIP

Unsafe Consumption of APIs was included in the OWASP top 10 list of API threats in 2023. Check out the official website for a brief description of this vulnerability: https://owasp.org/API-Security/editions/2023/en/0xaa-unsafe-consumption-of-apis/.

We’ll dive deeper into this topic in section 5.5, but for now suffice it to say that you must apply strong validation and sanitization controls on data coming from third-party APIs.

Figure 3.16 A threat actor updates their details with malicious code on an external identity provider. Later, they register with our website bringing in their details from the external service provider, and their malicious code runs against our database dropping the table users.

Now, what about data coming from our own internal services? Surely everything that comes from our system must be safe and valid? Not so fast! First, service to service integrations are complicated. As illustrated in figure 3.17, even a small mistake in the implementation of payload model can lead to cascading errors. For example, imagine service A checks the status of a payment with service B. If service A expects an enumeration with values “pending” or “paid” and service B sends returns “processing”, service A may break and the end user will get a confusing error. To prevent this situation, ensure you always validate data coming from other services and raise an appropriate error when you receive bad data. Ideally, raise an alert to notify the relevant team too.

Figure 3.17 When we don’t validate the responses from other services, we risk triggering cascading errors that result in confusing errors for our users.

All these problems come together in the response data flow. Every request to our API needs a response, and often that response contains data - except 204 responses which are empty.[9] As shown in figure 3.18, composing a response usually involves pulling data from our database, collating data from other services, third-party APIs, and finally, putting it all together into a single payload. This exposes the response flow to the third-party and the service-to-service flow, but there’s more. What about data coming from our own database? Can that be trusted? No, not really.

Figure 3.18 Composing a response’s payload often involves collating data from different sources, as in this example of a user requesting the status of their order. In this case, the orders service aggregates data from third-party APIs, other internal services, and its own database.

As represented in figure 3.19, there are many ways data can be altered in our database, including manual changes, accidental scripts, or the wrong migration. Remember, cybersecurity incidents are not a matter of if, but when, and we must be prepared to handle them. Make sure you validate the data you pull from the database using schema validation libraries or Object Relational Mappers (ORMs). ORMs help by creating a one-to-one mapping between our models and the database tables, and good ORMs help us validate constrains such as enumerations.

Figure 3.19 Our database can be corrupted in multiple ways, including accidental scripts, the wrong migration, or manual changes. If we fail to validate data from our database, we risk cascading and confusing errors in our applications.

What’s left in the response flow? As you can see outlined in figure 3.20, the final step involves putting together the data we’ve collected from various sources and assembling it into a single payload. Before sending the payload to the user, make sure you validate that it confirms to the right API schema. Validating the payload at this stage ensures you don’t send malformed data to the client and helps you raise meaningful errors when bad data leaks into the payload, hence giving you visibility and traceability of the problem.

Figure 3.20 Zero Trust APIs ensure consumers receive good data by validating all data sources regardless of their origin.

It’s clear that we must validate all data flows to safeguard the security and integrity of our systems. The impeding question is: what about internal APIs? Let’s tackle that in the next section.

3.4 No such thing as an internal API

Organizations build APIs for different purposes like offering products and services over APIs or to drive integrations between internal components. Another popular use for APIs is to automate internal processes, also known as back-office APIs because they’re not related to direct sales. Back-office APIs help manage staff and payroll, prepare sales forecasts, manage projects, and other internal processes. Due to their inherent nature, back-office APIs expose highly sensitive data about employees, customers, and other business and trade secrets. The bigger the organization, the more of this type of APIs.

Since they’re meant for internal use, back-office APIs are often assumed to be “internal APIs,” meaning they’re not public. The implementation details vary from organization to organization, but often, “internal APIs” run in a private network as illustrated in figure 3.21. The practice of breaking down a network into small subnets with varying degrees of access is called network segmentation, and it’s a very powerful security practice. But it shouldn’t be our only security strategy.

Figure 3.21 Organizations usually have public and internal APIs. Public APIs are available to all external users, while internal APIs are only available to employees. Internal APIs help automate internal jobs like customer and inventory management, and due to the sensitive data they contain, run in private networks.

Traditionally, cybersecurity teams have deemed internal APIs less risky since they run in private networks. Sadly, this means that back-office APIs often get very little love in the way of design and security efforts, and this makes them security ticking bombs. Internal APIs often suffer from lack of authorization and access controls, excessive data exposure, lack of documentation, and more. Consequently, if the API accidentally becomes publicly available due to a configuration or deployment mistake, our security posture is immediately compromised. How likely is this to happen? According to Salt Security, 8% of all attacks against APIs in 2023 were directed against internal APIs,[10] and we have reason to believe the trend will grow in the coming years.

This is what happened in September 2023 to Optus, Australia’s third largest telecommunications provider. According to an insider source, Optus created a customer identity API as part of a bigger project to enable two-factor authentication. The API was for internal use and was supposed to run in a private network, hence had no access controls whatsoever – no authentication or authorization required to access it. However, accidentally, the API got deployed to a “test network which happened to have internet access,” and millions of customer records were exposed.[11]

What can we learn from the Optus breach?

First, all APIs must be adequately protected with robust authentication and authorization, and the more sensitive the data the API exposes, the stricter the access controls.

The second lesson is that there is no such thing as an internal API. Whether an API gets deployed to a private network or not is an artificial distinction. There’s a risk of human mistakes when deploying APIs or configuring our networks, and we stand our best chances to prevent a data breach if we apply the Zero Trust Security model to all our APIs.

Internal APIs without proper access controls are also vulnerable to insider threats, whether intentional or accidental. An example of intentional insider threat is an insider malicious actor who steals or leaks sensitive data from your organization. An example of insider accidental threat is an overprivileged user who accidentally modifies or deletes sensitive data or leaks sensitive secrets by mistake. By applying the Zero Trust model to the design and implementation of internal APIs, we stand a better chance of mitigating these risks.

This is not to say that deploying internal APIs to a private network isn’t useful. Network segmentation is a very powerful security method, and anything that’s not supposed to be public shouldn’t be public, but it shouldn’t be our main or only strategy. In fact, as Kindervag highlights, “[n]etwork segmentation is a tactic and a tool, not a strategy for building secure networks.” To properly secure our APIs, we need something more fundamental approach - we need to shift left on our security and apply the principles described in this chapter. The remaining question here is: how do we know that we are protecting all our assets? Let’s tackle this question in the next section!

3.5 You can’t protect what you don’t know

We know we must protect all our APIs, including public and private APIs. But where are our APIs? One of the biggest risks organizations face in the API space is lack of documentation and inventory, which is part of a bigger problem known as API sprawl. API sprawl refers to the proliferation of APIs within organizations and their failure to track, document, and properly assess the security posture of those APIs. The API security industry has created catchy names for the types of APIs that emerge under API sprawl, such as Shadow APIs, Zombie APIs, and Ghost APIs.[12] Figure 3.22 illustrates the differences between these types of APIs, and the following paragraphs dive into them.

Figure 3.22 API sprawl raises the risk of shadow, zombie, and ghost APIs. These APIs tend to be less protected, and when hackers discover them, it can result in data breaches.

As organizations embark on Digital Transformation programs, cloud migrations, application modernization, and similar projects, they often need to publish APIs to expose new functionality and services. Unfortunately, many of these APIs are created ad-hoc, without proper planning, design, testing, and security assessments. These APIs created “under the radar” are known as Shadow APIs. Often, these APIs are undocumented, they are not part of a central catalog of APIs, and few members of the organization know about them.

DEFINITION

API sprawl is the proliferation of APIs without control, often duplicating existing functionality and without documentation.

DEFINITION

Shadow APIs are APIs built and released “under the radar”, without following a proper testing and evaluation process. These APIs tend to be undocumented, less secure, and quickly become forgotten, leaving a hole in the system. For a detailed overview of the security risks that shadow APIs pose for your organization, check out Teresa Pereira’s excellent talk “Do not live in the Shadow (APIs)”, delivered at APIDays Paris, December 5, 2024 (link to the recording: https://youtu.be/D2Gjf310QAs?si=ZZRFk_1u-sfEsSXy).

DEFINITION

Zombie APIs are deprecated APIs or versions of an API that we forgot to retire. Zombie APIs aren’t maintained anymore but they’re still exposed on the Internet.

DEFINITION

Ghost APIs are APIs exposed by our libraries or infrastructure. Ghost APIs aren’t a problem unless we don’t know about them and take appropriate measures to protect them.

Eventually, many shadow APIs are forgotten. They remain published, but nobody maintains them or looks after them. Forgotten APIs are known as zombie APIs. What about ghost APIs? Ghost APIs are APIs we haven’t created ourselves – they are developed by someone else, hence “ghostwritten.” For example, open-source libraries may expose endpoints that are not well documented or contain malicious dependencies that expose backdoors. In November 2020, the NPM security team removed from the repository a library called “twilio-npm” which exposed a TCP backdoor. See Pierluigi Paganini, “Malicious NPM library removed from the repository due to backdoor capabilities,” Security Affairs, November 3, 2020 (https://securityaffairs.com/110348/malware/npm-library-backdoor.html).

Equally, cloud services sometimes expose configuration and management endpoints. For example, AWS EC2 instances expose an Instance Metadata Service (IMDS) endpoint, which allows retrieving AWS access keys and other details. The IMDS is only available within the EC2 instance but can be exploited via an attack strategy called server-side request forgery (SSRF), which allows threat actors to trigger API calls to random endpoints from our system.[13]

How big is the scale of this problem? According to S&P’s The 2022 API Security Trends Report, organizations use 15,564 APIs on average, with large enterprises of more than 10,000 employees having 25,592 APIs on average. The number of APIs organizations use is also growing at an unprecedented speed, with an average growth rate of 201% in the 12 months leading to the survey. Many of those APIs are undocumented, and unsurprisingly, one of the top security concerns for organizations.[14]

Why is this a big problem? First, because shadow and zombie APIs often have a weaker security profile and hence compromise your security posture. And second, because without knowing how many APIs you are exposing, you don’t know what your attack surface looks like. As we saw in section 2.1, mapping out your attack surface is a pre-requisite for API security readiness and hence necessary to evaluate your API security posture.

How do you map out your attack surface and keep track of all your APIs? You need documentation. As Bruno Pedro highlights, there are different types of API documentation, such as specifications, tutorials, how-to guides, code samples, and changelogs.[15] If you offer products and services directly over APIs, it’s a good idea to document your APIs using all these strategies to deliver good developer experience and reduce the time to first call.[16]

The best tool for documenting your attack surface is an API specification. An API specification is a formal description of your API using a documentation standard, such as OpenAPI for REST APIs or the Schema Definition Language for GraphQL. API specifications are unambiguous and machine-readable, which means we can leverage them for automated testing, evaluation, and cataloguing purposes. If you don’t have standard specifications for your APIs, that’s the first problem you must tackle to improve your security posture.

API specifications are useful to understand how a single API works, but how do you manage an inventory of hundreds or even thousands of APIs? You need an API catalog. An API catalog is a tool that helps you track all your APIs in one place. With catalogs, you can browse and discover existing functionality within your organization. Why is that helpful? Because when you have 15,000 APIs in place and you’re thinking of building the next one, maybe you don’t need to – maybe it already exists already in the catalog. For large organizations, API catalogs are essential tools to fight API sprawl.

DEFINITION

An API catalog is a centralized and up-to-date list of all your APIs. Good API catalogs provide descriptions about the APIs and allow us to browse and look for existing functionality to avoid building duplicate APIs.

How do you build an API catalog? It can be as simple as a list of your existing APIs with their versions and descriptions. Traditionally, APIs.json (https://apisjson.org/) has been the reference in this space. APIs.json is an open specification for indexing APIs, similar to sitemap.xml for websites, and it supports human-readable elements like pricing, terms of service, and so on. More recently, Kevin Smith, senior technology strategist at Vodafone, authored an Internet-Draft proposing a standard location for API catalogs under a well-known URI - /.well-known/api-catalog (Kevin Smith, “api-catalog: A well-known URI to help discovery of APIs,” IETF Internet-Draft, May 4, 2023 [https://www.ietf.org/archive/id/draft-smith-api-catalog-00.html]).

“I’m not worried about what I know, I’m worried about what I don’t know.”

API discovery is the process by which we can discover attack surface exposed by our APIs that we didn’t know about. Discovery is one of the hottest topics in API security. Whenever I give a talk about API security, much of the conversation after the talk focuses on API discovery. Most of my previous consulting engagements included conversations about API discovery. As one executive put it once, “I’m not worried about what I know, I’m worried about what I don’t know.”

There are commercial solutions that implement API discovery with different approaches. For example, vendors like Traceable (https://www.traceable.ai/, now part of Harness) and Noname Security (acquired by Akamai, https://www.akamai.com/products/api-security) implement discovery through observability, while StackHawk implements discovery through code analysis. None of these solutions is perfect: discovery through observability only discovers what comes through in request logs. There may well be additional hidden endpoints that are not discovered because nobody uses them, until a hacker comes around, that is. Discovery through code analysis is incredibly challenging due to the many different ways endpoints can be implemented and exposed in a codebase, and is likely to offer mixed results unless it can go through every version of your software and determine when and how it was deployed.

The takeaway is that API discovery is very challenging, and even the most sophisticated solutions are likely to yield only partial results. As with most software problems, you can’t simply tool your way out of unknown APIs. The good news is, your operations and infrastructure teams probably know more than you think about what has been deployed when, where, and how. I suggest having conversations with your team before exploring commercial solutions. If anything, those conversations will get you in a better position to make the best use of commercial solutions.

To learn what’s possible in this space, check out Dan Gordon’s talk “API Catalog: The First Step in Protecting your APIs” at APIDays New York, July 27-28, 2022 (recording available on YouTube: https://youtu.be/ZMBN5muGtD4).

Of course, API catalogs make perfect sense when you have visibility of what APIs your organization is building and why. But what about those shadow APIs that your development team already deployed “under the radar”? For those APIs, you may want to explore commercial tools that inspect your API traffic to discover all your APIs and build an inventory on the fly. As illustrated in figure 3.23, this approach can be very effective for discovery shadow and zombie APIs when combined with an inventory of your known assets. The API catalog space is under active development as we speak, so make sure you research your options before making a choice.

Figure 3.23 API observability tools look at your traffic and discover undocumented attack surface by inspecting which requests are accepted by your API servers.

Once you’ve discovered your shadow APIs, the best move is to shift your API security to the left by planning and cataloguing your APIs before deploying them. You’ll need to align with your organization to ensure everybody understands why this is necessary (check section 2.5 for tips), and you’ll need a way to prevent unauthorized deployments. In chapter 9, you’ll learn about network segmentation techniques that allow you to disable unauthorized access to all your APIs by default and ensure inbound traffic only happens through well-documented and protected endpoints.

3.6 DevSecOps for APIs

We’ve learned a whole lot of things about shifting our API security to the left, Zero Trust security, the importance of validating all data flows and mapping our attack surface, and more. The question now is, how does it all come together in the software development life cycle? What does the actual process of building an API following shift-left security principles look like? In this section, we explore the role of security in the API development process as illustrated in figure 3.24. Figure 3.24 represents a model of the ideal SDLC for APIs. In the rest of this section, we dive into the details of each stage in figure 3.24.

Figure 3.24 Shift-left on security means we assess vulnerabilities at every stage of the development cycle and we don’t move on until we address them.

As shown in figure 3.25, it all begins with a design. At this stage, we focus on the capabilities we want to offer through the API, the type of data we’re willing to expose, and the user flows that will allow our customers to achieve their goals. This is a golden opportunity to identify sensitive operations that need extra security considerations. The result of this exercise is an API specification that describes our requirements in detail.

Figure 3.25 API design is a great opportunity to assess how vulnerable our user flows are and determine where we need to spend extra care from a security perspective. We write functional and security tests to reflect our expectations.

We’ll later use the API specification as an artifact to validate our implementation and assess our security posture, so it’s important that all relevant stakeholders have visibility of it and that we can track changes. Using a Git repository is a great way to accomplish both goals. If we later have to make changes to the API, the first step is changing the specification, and using Git it becomes as simple as raising a pull request (PR). All relevant stakeholders can come together in the PR to discuss and validate the change. As highlighted in figure 3.26, you can manage your API specifications in a common repository, or you can place the specification within the repository that implements the API. Personally, I find the latter approach very convenient, but a common repository may help get better visibility of all API developments and enforce the same standards.

Figure 3.26 Some organizations prefer a centralized repository with all API specs, which gives better visibility of all APIs and facilitates enforcing common guidelines. Others prefer to include the API spec in the application’s repo, which facilitates implementation testing against the spec.

Before we move on to the implementation, we test the design to identify vulnerabilities and address them – you’ll learn more about this type of API testing in chapter section 12 As leading APIOps expert Ikenna Nwaiwu highlights, this is also a great opportunity to write functional and security tests and to define essential KPIs for the API. This part of the process is essential to create a strong alignment with our stakeholders.

At this point, the API specification is simply a hypothesis of what we want to build. Once we start building the API and testing it, we’ll discover shortcomings in our original design, and hence our specification will inevitably change and evolve over time. My recommendation is to give your best shot to the design but not spend too much time on it since it’s going to change anyways. Also, don’t forget to version your APIs. At some point, you may need to introduce non-backwards compatible changes, and versioning helps making a smooth transition between versions. To learn about API versioning strategies, check out Jose Haro Peralta’s Microservice APIs, “Appendix B: Managing an API’s lifecycle.”

Definition

A mock server is a server that replicates the behavior of your APIs based on the specification. Popular tools to run mock servers are Prism (https://github.com/stoplightio/prism), WireMock (https://github.com/wiremock/wiremock), and Microcks (https://github.com/microcks/microcks). For more details about how mock servers work and how to use them, see chapter 7 of Jose Haro Peralta’s Microservice APIs (Manning, 2022).

With the API specification ready, it’s time to build. If the API will be consumed by a component within our platform, like a web or mobile application, or a microservice, we can work on the API server and and the consumer in parallel with the help of mock servers, as illustrated in figure 3.27. It’s also good practice to generate software development kits (SDKs) that make it easier to consume the API.[17]

Figure 3.27 Once we produce an API specification, we can work on the API client and the server in parallel. We simulate the backend using mock servers while we work on the client, and we validate the server implementation against the specification to ensure the integration works.

Remember, this is our opportunity to validate that our API design fits our needs and constraints. If we must make changes, we want to know as soon as possible, so shorten the feedback loop by deploying small changes frequently. As shown in figure 3.28, your continuous integration pipeline plays a crucial role here by testing your implementation before it gets deployed.

Figure 3.28 Our Continuous Integration and Delivery (CI/CD) pipeline plays a crucial role ensuring only reliable and secure code gets deployed. If any of our tests fail, we go back to the design stage to re-assess our API vulnerabilities.

You may want to run manual QA and security tests. Ideally, these are fine checks that are very difficult to automate and address vulnerabilities that are difficult to capture with automated tools. For example, if we have an ecommerce site, is our API vulnerable to scalping attacks, i.e. buying out the whole stock of a product to resell it at a higher price? Are threat actors able to scrape our whole catalog of products? Are we adequately protected against brute force authentication attacks? Like always, our API is vulnerable, and we must change the design, we should know as soon as possible, so run these tests frequently and automate them when possible. Finally, as shown in figure 3.29, when the API is good to go, we release it to production, and we publish it to our catalog.

Figure 3.29 If the final round of manual vulnerability assessment goes well, we move on the the final stage of the API development life cycle: publish to our API catalog and release.

This concludes our overview of the principles of API security. You’ve learned what API security by design is and how it helps us shift our security strategy to the left. You’ve learned to apply the Zero Trust Security model to APIs, where and how to validate in your data flows, and how to leverage documentation for vulnerability assessment. Finally, you’ve also learned to bring it all together into the software development life cycle by making vulnerability assessments at every stage. In the rest of the book, we’ll get into the nitty gritty of vulnerability assessment, the types of API vulnerabilities, and how to protect our applications against them.

3.7 Summary

Shift-left security is a paradigm that encourages us to address vulnerabilities at every stage of the software development life cycle (SDLC).

Tackling security early allows us to build security into our APIs. Tackling security at the end results in inefficient security solutions and more vulnerable applications.

Security by design enables shifting left on security by tackling vulnerabilities from our software’s design stage.

When we apply security by design to our APIs, we are less likely to find major vulnerabilities at the end of the SDLC, allowing us to release faster and with more confidence.

The Zero Trust Security model encourages us to remove trust from our systems and validate all traffic regardless of its origin.

Zero Trust APIs validate data across all flows, including third-party APIs, our own databases, internal services, the request and the response flows.

There is no such thing as an internal API. Zero Trust security means applying the same level of security to all our APIs.

Bad API documentation or lack thereof means we don’t know how many APIs we have or how they work.

API sprawling is the proliferation of duplicated and undocumented APIs. It includes:

Shadow APIs: undocumented APIs released “under the radar” with little or no checks.

Zombie APIs: deprecated API versions that we forgot to retire.

Ghost APIs: APIs exposed by our third-party libraries or systems.

API catalogs help us fight API sprawl by facilitating API discovery.

To shift left on API security, begin by assessing vulnerabilities during the design stage. Tackle security at every stage of the SDLC and address vulnerabilities when tests fail before moving on.

[1] Andrew Binder and Dan Barahona, “API Security 101: Establishing & Managing a Secure API Program,” webinar hosted by Redspin on February 11, 2021 (recording available on YouTube: https://youtu.be/cMN6yr4nBUo).

[2] Anand Prakash, “I Could Have Hacked All Uber Accounts- But I Chose to Report it Instead”, HackerNoon, September 13, 2019 (https://hackernoon.com/how-i-could-have-hacked-all-uber-accounts-rtzl3z72).

[3] S&P Global, 2022 API Security Report (https://nonamesecurity.com/wp-content/uploads/2023/06/Whitepaper-451-Research-Trends-Report.pdf), pp. 11-12.

[4] John Kindervag, “Build Security Into Your Network’s DNA: The Zero Trust Network Architecture,” Forrester Research (November 5, 2010) and “No More Chewy Centers: The Zero Trust Model Of Information Security,” Forrester Research (March 23, 2016).

[5] Scott Rose, Stu Mitchell and Sean Connelly, “Zero Trust Architecture,” NIST Special Publication 800-207, August 2020 (https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf).

[6] Eric Newcomer, “API Security: Is Authorization the Biggest Threat?” The New Stack (June 12, 2023, https://thenewstack.io/api-security-is-authorization-the-biggest-threat/).

[7] Salt Security, State of API Security Q1 2023, p. 4.

[8] Aner Morag, “Coinbase Fixes Vulnerable API that Let You Sell Bitcoin You Didn’t Own,” Security Boulevard, February 20, 2022 (https://securityboulevard.com/2022/02/coinbase-fixes-vulnerable-api-that-let-you-sell-bitcoin-you-didnt-own/).

[9] For an overview of the most common types of responses in APIs, see section 4.6 of my book Microservice APIs (Manning, 2022).

[10] Salt Security, State of API Security Q1 2023, p. 4 (https://content.salt.security/rs/352-UXR-417/images/SaltSecurity-Report-State_of_API_Security.pdf).

[11] Andrew Greene, “Optus rejects insider claims of 'human error' as possible factor in hack affecting millions of Australians,” ABC News, September 23, 2022 (https://www.abc.net.au/news/2022-09-23/optus-rejects-claim-hack-likely-result-of-human-error/101468846).

[12] For an excellent overview of this problem, check Nick Rago’s article “Are You Haunted by Zombie, Shadow and Ghost APIs?” Salt Security Blog, October 28, 2022 (https://salt.security/blog/are-you-haunted-by-zombie-shadow-and-ghost-apis).

[13] Nick Frichette, “Steal EC2 Metadata Credentials via SSRF,” Hacking The Cloud, August 1, 2020 (https://hackingthe.cloud/aws/exploitation/ec2-metadata-ssrf/).

[14] Salt Security, State of API Security Q1 2023 (https://content.salt.security/rs/352-UXR-417/images/SaltSecurity-Report-State_of_API_Security.pdf). 36% cited Zombie APIs as a major security pain point and 34% cited shadow APIs too.

[15] Bruno Pedro, “Five Elements of Good API Documentation,” The API Changelog, December 8, 2023 (https://apichangelog.substack.com/p/five-elements-of-good-api-documentation).

[16] Joyce Lin, “The most important API metric is time to first call,” TechCrunch, July 12, 2021 (https://techcrunch.com/2021/07/12/the-most-important-api-metric-is-time-to-first-call).

[17] To learn about automatic SDK generation, check out Sidney Maestre’s excellent talk “You Don’t Need SDKs, Wait Maybe You Do?” at the Platform Summit 2023, October 16-18, Stockholm (recording available on YouTube: https://youtu.be/ezEQ5ewLP4A).

4 Top API authentication and authorization vulnerabilities

This chapter covers

Mitigating API authentication and authorization vulnerabilities

Common flaws in role-based access controls

Preventing unintended updates to our data

Mitigating sensitive data leakage

Preventing abuse of our business logic

APIs expose access to sensitive data and operations in our systems and it’s clear that we must protect them. But what are we protecting them against? What do API vulnerabilities look like and how are they exploited? How do we defend our APIs against those vulnerabilities? How and when do we know that we’ve done enough to mitigate the risks to our APIs? If these questions are bugging you, you’ve come to the right place. In this chapter, we examine the OWASP Top 10 API Security threats and we learn to mitigate them.

OWASP is a nonprofit organization created by Mark Curphey in 2001 to support a host of community-driven projects in cybersecurity. In 2003, OWASP launched its signature Top 10 Web Application Security Risks, which has since been regularly updated, with the latest list published in 2023 (https://owasp.org/www-project-top-ten/). As a developer, OWASP is your first stop to understand what kinds of threats your applications are facing and how to deal with them.

APIs didn’t catch the attention of OWASP until recently. That’s understandable – we’ve been building APIs for decades, but it wasn’t until recently that APIs came into the spotlight as a security liability. OWASP published its first list of top 10 API security threats in 2019 (https://owasp.org/API-Security/editions/2019/en/0x00-header/), and ever since it’s been a reference for API security practitioners. OWASP continues to monitor the API security landscape to keep the list of top 10 threats up to date, and they made a major update in 2023 (https://owasp.org/API-Security/editions/2023/en/0x00-header/).

The publication of OWASP’s API Security Top 10 was a turning point in the API space. It acknowledged that API security is different from traditional web security and that it brings new challenges. But more importantly, it gave us all an overview of the main types of threats APIs are facing. In this chapter and the next, we analyze the OWASP top 10 API vulnerabilities with specific examples of what they look like, how threat actors exploit them, and how we mitigate them. In this chapter, we look at vulnerabilities that have to do with authentication and authorization access controls in our API, and in the next chapter we look at API configuration and management-related vulnerabilities.

Whenever possible, I illustrate every vulnerability with a coding example, and also show practical strategies to address and mitigate the vulnerabilities. The coding examples aim to demonstrate how the vulnerabilities become exploitable, and to that extent, I have made a conscious effort to keep the listings simple. My goal is to help you understand the basic patterns that lead to vulnerabilities and exploits in your APIs, and if you study the examples carefully, you’ll be able to spot these patterns in your own APIs too.

The examples are written in Python, but they are generic enough to apply to all development stacks. When illustrating a vulnerability, the coding examples focus on the code that demonstrates the vulnerability. If you want to run the vulnerable server and play around with it, you need additional bootstrapping code to set up the server; you can find that code in this book’s GitHub repository for this chapter (http://github.com/abunuwas/secure-apis/ch04).

4.1 Running the code examples

Chapters 4 and 5 contain plenty of practical coding examples that illustrate the types vulnerabilities discussed throughout the chapters. I’ve also included coding examples of the solutions needed to address and mitigate the vulnerabilities. If you want to try out the code examples from the book’s GitHub repository as you read through the chapters, please read this section before you proceed to understand how they set up and how to work with them.

When looking at the code listings in chapters 4 and 5, note that the listings only show the code fragments that are relevant to illustrate the vulnerabilities and their remediations. The full implementations are available in the book’s GitHub repository (https://github.com/abunuwas/secure-apis), including additional bootstrapping code needed to set up the API server, the database, seed data, and so on. I’ve excluded the bootstrapping code from the code listings since it doesn’t add value to the explanations, but it’s available in the book’s repository if you want to understand how it works.

To run the code examples for chapters 4 and 5, make sure you have Python 3.11 or later. If you don’t have Python in your machine, follow the instructions on the Python Foundation’s website to install it (https://www.python.org/downloads/). The code examples use the uv dependency manger, one of Python’s most popular dependency management tools, to install third-party libraries. Once you have a Python runtime, install the uv dependency manager using the following command:

pip install uv

Note that depending on your Python installation configuration, the pip command may take different forms like pip3, pip3.11, pip-3.11, etc., with the suffix reflecting the version of your Python runtime.

Once you have uv, you can install and manage additional Python versions using uv. Follow the official uv documentation for additional information on that feature (https://docs.astral.sh/uv/concepts/python-versions/).

All the code listings in chapters 4 and 5 use the same dependencies. The full list of dependencies is listed in the ch04/pyproject.toml file, which contains additional configuration about the project, such as its name, version, and author. I’ve selected FastAPI (https://github.com/fastapi/fastapi), which is Python’s most popular framework for building REST API applications, to build the APIs, and SQLAlchemy (https://github.com/sqlalchemy/sqlalchemy), which is Python’s most popular object relational mapper (ORM), to interface with the database. If you want to learn more about how FastAPI and SQLAlchemy work, check out chapters 2, 6, and 7 of Jose Haro Peralta’s Microservice APIs (Manning, 2022).

To install the dependencies needed to run the code, use the following command:

uv sync

This creates a local virtual environment under a folder named .venv/ with all the dependencies. After installing the dependencies, activate the virtual environment with the following command:

source .venv/bin/activate

You are now ready to run all the code examples. Every example features an API server that illustrates the vulnerabilities discussed in the following sections. For example, to run the server with the BOLA vulnerability, you use the following command:

uvicorn bola:server --reload

The above command uses uvicorn (https://github.com/encode/uvicorn), which is a web server commonly used to run FastAPI applications. The server runs in the localhost on port 8000, and you can access the auto-generated API’s Swagger here on the following URL: http://localhost:8000/docs. You’ll find it easier to interact with the APIs using that URL.

Some endpoints in the examples are authenticated, and to interact with them you need a valid access token. The ch04/auth.py module implements authentication access controls for all the servers, and it expects tokens from the apithreats.com website. To obtain a valid access token, use the following URL: https://apithreats.com/login.

One feature of the practical code examples in chapters 4 and 5 is that they come in two steps: first, we encounter the vulnerable version of the API, and then, we change the code to fix the vulnerability. In the book’s GitHub repository, you encounter the two versions of the code, with the vulnerable version of the API URL path. For example, for the BOLA illustration, you’ll find the vulnerable implementation under GET /vulnerable/orders/{order_id} and the safe version under GET /orders/{order_id}. The same pattern is replicated across all other examples, and the ch04/README.md and the ch05/README.md files contain full details on each server implementation and where to find the vulnerable and the safe endpoints.

4.2 Broken object-level authorization

Broken object-level authorization (BOLA) happens when an API fails to apply user-based access controls to resources. For example, when user A can access resources or operations that should only be accessible to user B. Let’s use an example to illustrate this concept. Figure 4.1 illustrates a simple website platform where users can publish and edit their content. Publishing happens through the POST /posts endpoint and editing through the PUT /posts/{post_id} endpoint. Editing a post on the PUT /posts/{post_id} endpoint should only be available to the post’s author. For instance, if user B publishes a post and the post’s ID is 1, then the PUT /posts/1 resource should only be available to user B. However, as shown in figure 4.1, when user B publishes a post on the blogging website, user A can edit it, which means the website is failing to enforce user-based access controls and therefore is vulnerable to BOLA.

NOTE

Broken Object Level Authorization (BOLA) is also often called insecure direct object reference (IDOR). The idea behind IDOR is the same: threat actors can access resources that don’t belong to them. BOLA is a more recent nomenclature, and generally, you’ll see IDOR more frequently used in older publications.

Figure 4.1 BOLA happens when user B can access resources or operations that should only be accessible to user A, like updating user A’s posts.

BOLA is a leading cause of security incidents in APIs. In November 2018, the United States Postal Service (USPS) fixed a vulnerability that allowed any authenticated user to access personal details of other users through its “Informed Visibility” API. Informed Visibility was a real-time tracking service for USPS business partners. The API also gave access to user’s personal details, such as their email addresses, phone numbers, and personal addresses, and allowed threat actors to make changes to them. Unfortunately, any authenticated user (i.e. any user with an account) could access the Informed Visibility API and hence query the system to obtain personal details of other users. For more information on this breach, see Brian Krebs’s article “USPS Site Exposed Data on 60 Million Users,” (KrebsonSecurity, November 21, 2018, https://krebsonsecurity.com/2018/11/usps-site-exposed-data-on-60-million-users/).

In sensitive sectors like healthcare, finance, and defense, BOLA is a leading factor in data breaches. For example, when I make payments with my bank, I should be the only user who can access information about those payments. If the bank’s API is vulnerable to BOLA, other users will be able to access information about my payments, and that will result in a data breach.

Sadly, businesses operating in sensitive data sectors are not immune to BOLA either. For example, in February 2024, fertility tracker Glow addressed a vulnerability in its API that allowed any authenticated user to access personal data from other users in the platform, including their age, address, and user-uploaded images. For more details on this breach, see Lorenzo Franceschi-Bicchierai, “Fertility tracker Glow fixes bug that exposed users’ personal data,” (TechCrunch, February 13, 2024, https://techcrunch.com/2024/02/13/fertility-tracker-glow-fixes-bugs-that-exposed-users-personal-data).

How does BOLA happen and what can we do to prevent it? BOLA occurs at the resource level. As illustrated in figure 4.2, most APIs have a concept of resource. For example, in an e-commerce application, we may have multiple resources that represent things like payments, products, orders, and more. When we place an order, we create an order resource.

Figure 4.2 Most APIs have a concept of resources. For example, a REST e-commerce API may have a concept of product resources served under a /products endpoint, order resources under /order, and so on. To prevent unauthorized access, we must protect each resource with the right access controls.

Every resource has an ownership and permissions models that determine the access controls on the resource. The more sensitive the resource, the stricter the access controls. For example, orders and payment data are very sensitive, so when I place an order, I’m the owner of that resource and the only user allowed to view, edit, or cancel it. Meanwhile, user reviews and product descriptions are less sensitive, so everybody can see them, but only the owner can edit or delete them. In this case, BOLA happens if a threat actor manages to see, edit, or cancel my reviews.

How can threat actors get their hands on my orders? Every resource has an identifier. For example, in a REST API, an order may be represented through a URL like /orders/1 if the order’s identifier is numerical, while a GraphQL API may expose a query operator like getOrder(id: ID!) that allows us to retrieve the details of an order. In either case, threat actors will play with different order ID values to try and get access to resources that don’t belong to them. The key to preventing BOLA is ensuring we have an accurate implementation of our access controls to every resource. Let’s see how we do that with a specific example.

4.2.1 Practical example of BOLA

From an implementation point of view, BOLA occurs when we fail to check that the right user is accessing the requested resource. For example, in an e-commerce website, we may have an API to place orders and retrieve their details. Since orders contain sensitive information, including user address and payments, we want to restrict access to their legitimate owners. In this case, BOLA happens when user B can access order details from user A. Listing 4.1 shows an example of that.

Listing 4.1 implements an endpoint GET /orders/{order_id} that allows us to fetch the details of an order. The endpoint is authenticated and therefore only accessible with a valid token. The function validate_access() validates the token, and if it’s valid, it returns an instance of the UserClaims object, which is assigned to the user_claims parameter. Token validation happens automatically through the use of the Depends() function, which is FastAPI’s implementation of dependency injection.

In line 7, we open a database session to run a query and search for the requested order. We run the query using Python’s popular ORM SQLAlchemy, and the only condition we add to the query is that the record’s ID matches the ID requested by the user. The query in line 9 translates into the following SQL statement: SELECT * FROM TABLE order WHERE order_id = @order_id;. If we don’t find any records matching the ID, we return a 404 response, and otherwise we return the full order details. The 404 (Not Found) status code means that the resource requested could not be found in the server, hence it’s an appropriate status code for this situation.

Listing 4.1 Implementation of an orders resource endpoint vulnerable to BOLA

# file: ch04/bola.py
@server.get("/orders/{order_id}")    #A
def get_order_details(
        order_id: int,    #B
        user_claims: UserClaims = Depends(validate_access)    #C
):
    with session_maker() as session:    #D
        order = session.scalar(    #E
            select(OrderModel).where(OrderModel.id == order_id)
        )
        if order is None:    #F
            raise HTTPException(
                status_code=404,
                detail=f"Order with ID {order_id} not found."
            )
    return {    #G
        "id": order.id,
        "product": order.product,
        "quantity": order.quantity,
    }

So, how is listing 4.1 vulnerable to BOLA? The code doesn’t check who’s requesting the order details, so anyone with a list of valid order IDs can access other users’ orders. You can confirm this by running a manual test. Follow the instructions in section 4.1 to create a virtual environment and install the dependencies, and run the following command to start up the server:

uvicorn bola:server --reload

Login using https://apithreats.com/login to obtain an access token (follow the instructions under ch04/README.md for more details on this), and send the following request to the server:

curl 'http://localhost:8000/orders/1' -H 'Authorization: Bearer <access_token>'

If you’re running the code directly from the book’s GitHub repository, you can find the vulnerability under GET /vulnerable/orders/{order_id} endpoint. To prevent this vulnerability, we need to check if the right user is sending the request. Listing 4.2 adds a second condition to our database query to check that the requested record also matches the user’s ID. With this change, the query statement becomes SELECT * FROM TABLE order WHERE order_id = @order_id and user = @user_id; Thanks to this change, threat actors won’t be able to access other users’ orders by simply supplying valid order IDs. In section 7.3, you’ll learn more about token validation and extracting user details from the token to make them available to our routing functions.

Listing 4.2 Fixing BOLA vulnerabilities by checking a user’s right to access a resource

# file: ch04/bola.py
@server.get("/orders/{order_id}")
def get_order_details(
        order_id: int,
        user_claims: UserClaims = Depends(validate_access)
):
    with session_maker() as session:
        order = session.scalar(
            select(OrderModel).where(
                OrderModel.id == order_id,
                OrderModel.user == user_claims.sub,    #A
            )
        )
        if order is None:
            raise HTTPException(
                status_code=404,
                detail=f"Order with ID {order_id} not found."
            )
    return {
        "id": order.id,
        "product": order.product,
        "quantity": order.quantity,
    }

In listing 4.2, when a threat actor requests orders from other users, we respond with a 404 (Not Found) status code, which means the resource was not found. This response is fitting in this situation, since we don’t want to reveal whether those resources really exist or not. If the resource is known to the user and they just happen to lack access to it, we can respond with a 403 (Forbidden) status code.

The 403 (Forbidden) status code signals that the user has valid credentials but doesn’t have the right permissions to access the requested resource. Therefore, 403 responses implicitly acknowledge that the resource exists. 403 are perfectly fine in some situations. For example, in a project management application, it’s OK to reveal to a user that a ticket exists within their organization, but they don’t have access to it. This feedback is useful and actionable so that the user can reach out to the relevant person within their team and get access to the ticket. For an excellent analysis of the security implications of using different status codes, check out Tereas Pereira’s “How can HTTP status codes tip off a hacker?”(Cyber Security News, November 19, 2024, https://cybersecuritynews.com/how-can-http-status-codes-tip-off-a-hacker/).

Another subtle consideration is response times. If your API consistently responds with different latencies to requests for existing and non-existent resources, a sophisticated threat actor can leverage that information to enumerate resources. This is known as a side-channel attack, since the threat actor is inadvertently leaked by the system. Check if your API follows such pattern, and if it does, consider putting measures in place to deliver consistent response times across all requests.

DEFINITION

A side-channel attack is a type of attack where the threat actor leverages information inadvertently leaked by our system to gain knowledge about internal configuration, discovery data, and more. One example of side-channel attack is a timing attack. In the case of APIs, timing attacks exploit request-response latencies to gain knowledge about our application via resource enumeration and other types of exploits.

BOLA is a widespread vulnerability in the API space, and it only takes a vulnerable endpoint to end up in a data breach. To mitigate this risk, review and test all your API endpoints to see if any of them are vulnerable to BOLA. You can begin by making the type of manual test illustrated in this section, and in chapter 12, you’ll learn to automate BOLA tests. If you discover that an endpoint is vulnerable to BOLA, you must prioritize this issue to fix it as soon as possible following the recommendations in this section.

A strong authorization model builds on top of robust authentication, and sadly, this is one area where many APIs fail to get it right. In the next section, we look at the most common flaws in API authentication and how to address them.

4.3 Broken authentication

Most APIs need an authentication system. As illustrated in figure 4.3, authentication systems allow us to manage user accounts, verify the identity of a user when they log into our website, and issue access tokens. Authentication is the most critical component of our API security strategy and underpins our ability to protect our APIs. Many of the security measures we discuss in this chapter rely on the information contained in access tokens. If threat actors can easily take over other user accounts and steal their tokens, the rest of our security strategy won’t work.

Figure 4.3 An authentication system manages users’ identities, credentials, and access permissions. When a user logs in, they get back an API access token with specific permissions.

How does broken authentication happen? We’re looking at vulnerabilities that allow threat actors to hijack our authentication and authorization flows, break into user accounts, and tamper access tokens. What makes authentication and authorization flows vulnerable? As illustrated in figure 4.4, a common cause is poor credentials management. For example, many websites have weak password policies that allow users to employ easily guessable passwords.

Figure 4.4 Credential stuffing is a common exploit in which threat actors use leaked user credentials from other websites and insert (aka “stuff”) them in our API to try and take over user accounts.

Equally prevalent is the lack of prevention against brute force attacks, allowing threat actors to attempt multiple combinations of username and password until they find a matching combination. Another common attack strategy is credential stuffing. Users often tend to use the same credentials across multiple websites, which means if their credentials are leaked from one site, threat actors will insert, or “stuff” them into other websites to try and take over other user accounts.

A recent trend in login flows involves introducing the username or email in the first step, and if the username or email isn’t registered, prompting the user to create an account, and otherwise proceeding with their password. Gmail uses this approach, and many other websites are following suit. The problem with this approach is that, without excellent threat detection support, it represents an excellent opportunity for threat actors to figure out which user accounts are registered. In other words, multi-step login flows may offer a chance to enumerate users registered on our website.

Other common vulnerabilities include exposing sensitive credentials, like passwords or access tokens, in the URL, and allowing users to change sensitive details, such as their username, email, or password, without password confirmation, which allows threat actors to modify them after hijacking a user session or their access. Token.

NOTE

Why is it risky to expose sensitive credentials through the URL? There are two reasons: 1) URLs get recorded in the browser history, so if the API is accessed from a web application, a threat actor who gets access to another user’s browser history can get hold the user’s credentials; 2) another reason is because URLs end up in access log files (i.e. your server logs), and hence may be accessible to anyone who can get their hands on those files, including threat actors. How realistic is the probability that threat actors will ever break into our logs? More than you think. On October 2023, Okta suffered a breach that allowed threat actors to access HAR files from Okta customers, leaking all sorts of sensitive data and credentials. Check out Kenny Johnson’s overview of the breach in “Introducing HAR sanitizer: secure HAR sharing,” The Cloudflare Blog, October 26, 2023 (https://blog.cloudflare.com/introducing-har-sanitizer-secure-har-sharing/). HAR stands for HTTP Archive format, and it’s a standard for logging browser activity to a file in JSON format. Most browsers allow you to record traffic while you interact with a website and download the logs as a HAR file.

What about access tokens? As shown in figure 4.5, here we are looking at vulnerabilities in the process of issuing and validating access tokens, which can lead to token tampering attacks. Another type of vulnerability is credentials replay, in which a threat actor intercepts another user’s credentials, such as their access token, and uses them to impersonate the user. Credential replay is particularly concerning in highly sensitive applications such as those in banking, finance, healthcare, etc. In chapter 10, you’ll learn strategies to add proof of token ownership so you can prevent credential replay attacks.

Let’s illustrate how threat actors may exploit vulnerabilities in the credential validation process. The most common type of credential used in APIs is the JSON Web Token (JWT) specification. JWTs are JSON objects that contain information about the user and are signed to prove their integrity (see section 7.2 to learn more about JWTs). As illustrated in figure 4.5, two common vulnerabilities are failure to validate the token’s signature correctly, and failure to reject expired tokens. Failing to validate the token’s signature allows threat actors to tamper tokens, which means they can hijack other user accounts and potentially escalate their privileges.

Figure 4.5 A common broken authentication vulnerability is accepting unsigned or expired tokens.

Despite the crucial role that authentication plays in our systems, broken authentication vulnerabilities are widespread. Authentication is hard, and even the best among us get it wrong from time to time. For example, in 2014, Egor Homakov found vulnerabilities in the implementation of Open Authorization (OAuth) flows on GitHub that allowed him to gain access to other users’ Gists, repositories, and account details. OAuth is the industry standard for granting access to an API, and we’ll learn more about it in chapter 6. GitHub’s vulnerabilities included lack of validation of redirect URIs and the ability to request arbitrary scopes. Check out Egor’s post “How I hacked GitHub again” for a detailed breakdown of GitHub’s vulnerabilities (https://homakov.blogspot.com/2014/02/how-i-hacked-github-again.html).

Implementing OAuth flows is difficult; however, we find the bulk of security incidents in access token validation. For example, in December 2020, cybersecurity researcher Ron Chan discovered that Microsoft’s Outlook was accepting requests from unsigned tokens, which essentially would allow a threat actor to access the emails from any other account by sending tokens without signatures. Ron Chan documented the vulnerability in detail in a video titled “I hacked Outlook and could’ve read all your EMAILS!” published on YouTube on December 18, 2020 (https://youtu.be/t54N4x2uIPs).

As we mentioned earlier, the most common type of authentication credential in APIs is the JWT, and a common source of vulnerabilities in JWTs involves manipulating the token’s alg field. The alg field tells us what signature was used to sign the token, and according to the JSON Web Token (JWT) specification, it's perfectly fine to set the alg field to none, which essentially produces an unsigned token (https://datatracker.ietf.org/doc/html/rfc7519#section-6.1). OAuth best practices recommend always using signed tokens (https://datatracker.ietf.org/doc/html/draft-ietf-oauth-security-topics#section-2.5), and many sites explicitly prohibit tokens with the “alg” field set to “none.” However, the devil is in the details. In April 2020, Ben Knight from CyberCX discovered that Auth0 was banning tokens with alg set to none, but it was a case-sensitive check. Knight was able to forge Auth0 tokens by capitalizing one of the letters in none, for example by setting the token’s alg field to nonE.[1]

There are many more examples, and if you browse through HackerOne, a popular bug hunting website, you’ll find plenty of reports of broken authentication. You get the idea: broken authentication is prevalent and dangerous. So, what can we do to prevent it? The first thing I always recommend is don’t do it yourself. Authentication is a minefield, and as we saw with the GitHub’s example, even simple mistakes in the implementation of OAuth flaws can result in a major breach.

Instead of rolling out your own system, consider using identity as a service providers (IDaaS) like Okta, Auth0, Authlete, Curity, Microsoft Entra ID (formerly Azure’s Active Directory), and so on. Those organizations employ hundreds of developers and researchers entirely dedicated to monitoring and preventing authentication threats. That still leaves you with a few tasks, like integrating correctly with those services, using the right tokens for access control, selecting strong signing algorithms, and validating the tokens correctly. You’ll learn best practices to handle all these tasks in chapters 6 and 7, but to give you a taste of things, we’ll look at an example of broken authentication in the next section.

4.3.1 Practical example of broken authentication

Let’s see what broken authentication looks like in practice with an example of vulnerable access token validation. We’ll use JWCrypto (https://github.com/latchset/jwcrypto), a popular Python library for issuing and validation JWTs. There are other popular JWT libraries in Python, like PyJWT (https://github.com/jpadilla/pyjwt) and python-jose (https://github.com/mpdavis/python-jose), but JWCrypto gives us a degree of flexibility in how to validate tokens that can become a vulnerability if not properly understood, hence it becomes a great example of common broken authentication flaws.

Listing 4.3 shows a vulnerable implementation that fails to validate the token’s audience (aud) claim. The audience (aud) represents the API the token was issued for. To validate the token, we create an instance of JWCrypto’s JWT class, passing in the public key needed to validate the token’s signature, the token itself, and the algorithm used to sign it. If the token is valid, we return its claims. If the token is invalid, JWCrypto raises a JWTInvalidClaimValue error, which we capture to return an error response.

Listing 4.3 Vulnerable JWT validation with JWCrypto

# file: broken_authentication.py
def validate_token(token):
    try:    #A
        unverified_header = json.loads(
            base64.urlsafe_b64decode(credentials.credentials.split(".")[0])
        )    #B
        key = jwk.JWK(**find_public_key(unverified_header["kid"]))    #C
        valid_token = jwt.JWT(    #D
            key=key, jwt=credentials.credentials, algs=["RS256"],
        )
        return valid_token.claims    #E
    except (
        JWTInvalidClaimValue,
    ) as error:
        raise HTTPException(status_code=401, detail=str(error))    #F

Check out ch04/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and how to obtain an access token to test the vulnerability. Listing 4.3 is vulnerable because it doesn’t tell JWCrypto that it must validate the token’s audience. Checking the audience is important if we have different APIs. For example, a ride-sharing application may expose an API for drivers, another one for customers, and yet another one for administrators. Failure to check the audience, as in listing 4.3, means a normal customer could access admin functions. To prevent these vulnerabilities, we must tell JWCrypto to validate those claims. Listing 4.4 tells JWCrypto that the expected audience is https://apithreats.com/admin.

Listing 4.4 Vulnerable JWT validation with JWCrypto

# file: broken_authentication.py
def validate_token(token):
    try:
        unverified_header = json.loads(
            base64.urlsafe_b64decode(credentials.credentials.split(".")[0])
        )
        key = jwk.JWK(**find_public_key(unverified_header["kid"]))
        valid_token = jwt.JWT(
            key=key,
            jwt=credentials.credentials,
            algs=["RS256"],
            check_claims={
                "aud": "https://apithreats.com/admin",    #A
            }
        )
        return valid_token.claims
    except (
        JWTInvalidClaimValue,
    ) as error:
        raise HTTPException(status_code=401, detail=str(error))

Once we’ve got authentication under control, we can lay out the rest of our API security strategy. Our next stop is another very common vulnerability: Broken Object Property Level Authorization.

4.4 Broken object property level authorization

Broken object property level authorization (BOPLA) is a new threat category introduced in 2023 that brings together two categories from 2019: mass assignment (API6:2019) and excessive data exposure (API32019). Mass assignment happens when users manage to override the value of data they shouldn’t be able to change, and excessive data when we expose unnecessary amounts of sensitive data. Since the mechanics of both threats are different and the attack strategies also differ, we’ll treat them separately, beginning with mass assignment.

4.4.1 Mass assignment

Most applications have a concept of server-side data. Server-side data is data that should only be changed by our system or by authorized individuals such as administrators. For example, when we make a payment, our payments provider must audit the transaction to ensure it doesn’t involve fraud or crime and that we have the required funds. This means our payment goes through several stages like initiation, auditing, processing, and settlement. The payment can also be rejected if we have insufficient funds.

As illustrated in figure 4.6, these stages are reflected in the payments data model through a status attribute. A payments API allows us to make payments and check their status, but it shouldn’t allow us to change the status of a payment. For example, a user shouldn’t be able to make a payment and set its status to approved or settled right away. If that happens, it means our API is vulnerable to mass assignment.

Figure 4.6 A payment goes through a lifecycle with stages such as pending, accepted, settled, and rejected. Those stages can be documented as an enumeration in an OpenAPI schema.

As shown in figure 4.7, mass assignment happens when we blindly trust the data a user sends in a request and bind it directly to our database models. This allows threat actors to override sensitive and business-critical data in our systems, including the status of an order or payment, their loyalty points and discounts, and even their access privileges. In other words, mass assignment vulnerabilities can compromise the integrity of our system and our business.

Figure 4.7 Mass assignment happens when a threat actor manages to override a field they shouldn’ve have access to, like the status of a payment.

How bad does this get in real life? Let’s review some examples. In March 2012, Egor Homakov alerted the Ruby on Rails development team that the framework made applications built with it vulnerable to mass assignment attacks (https://github.com/rails/rails/issues/5228). Egor’s warning didn’t bring much attention, so he decided to prove his claim. Knowing that GitHub was built using Ruby on Rails, he performed the most famous mass assignment attack in history: he managed to commit to the master branch of Ruby on Rails’ official repository on GitHub. The commit is still visible on Ruby on Rail’s history (https://github.com/rails/rails/commit/b83965785db1eec019edf1fc272b1aa393e6dc57).

Homakov didn’t have the right to commit directly to the master branch, so how did he manage to do it? Homakov exploited a mass assignment vulnerability on GitHub’s API to bind his own SSH key to the profile of another user with commit rights to the Ruby on Rails’ repository.[2] Once he was in possession of an SSH key with commit access, he pushed his changes directly to the master branch. Thankfully, Homakov’s commit was harmless and only intended to bring attention to mass assignment vulnerabilities on websites built with Ruby on Rails.

Mass assignment was originally designed as a feature of the Ruby on Rails framework to save developers work when populating instances of active record objects. For example, when creating a new user, we could use the strategy shown in listing 4.5 to populate user attributes in one go, without having to set each value individually. In listing 4.5, :user represents the payload sent to the server to register a new user.

Listing 4.5 Example of mass assignment in Ruby on Rails

def signup
  params[:user] # => {:name => "John", :email => "john@example.com"}
  @user = User.new(params[:user])
end

The problem is, when exposed directly through the controller, as in listing 4.5, a threat actor can set arbitrary attributes in the payload and bind them to their user profile. For example, an attacker could send the following request to create an admin profile: http://www.example.com/user/signup?user[name]=John&email=john@example.com&user[admin]=1. This is also a great example of an elevation of privilege attack, whereby threat actors attempt to elevate their access privileges. To counter mass assignment vulnerabilities, Ruby on Rails introduced the concept of protected attributes, which allows us to declare properties that shouldn’t be bound directly to an active record object via mass assignment.

Despite growing awareness about the risks of mass assignment, many codebases are still vulnerable to it. You can find multiple examples of mass assignment reports on HackerOne. For example, in May 2017, a HackerOne user reported a vulnerability on Radancy NL, a Dutch company specialized on employer branding and recruitment advertising (https://hackerone.com/reports/227522; if you have trouble accessing the report, you can also check the details on this website: https://vulners.com/hackerone/H1:217558). The vulnerability allowed the user to take over resources belonging to other users by sending a POST request to edit the resource, including setting the owner of the resource to his user ID. Of course, mass assignment wasn’t the only problem here – the API wasn’t even checking if the request was sent by the resource’s legitimate owner, which means the API was also vulnerable to BOLA.

Mass assignment isn’t limited to Ruby on Rails. Any stack is vulnerable to mass assignment as long as you bind user data to your database models without validating and sanitizing it first. For example, listing 4.6 shows a code snippet in Python vulnerable to mass assignment. The listing implements an endpoint that allows us to make payments. As you can see in lines 9 and 10, we load the data sent by the user and bind it directly to our database model without validating and sanitizing it first. This allows threat actors to manipulate the status of their payments. For example, the following request makes a payment and sets its status to accepted: curl -X 'POST' 'http://localhost:8000/payments' -d '{"amount": 1000000, "currency": "USD", "status": "processed"}'. Check out ch04/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and run the previous command to test the vulnerability.

Listing 4.6 Python code vulnerable to mass assignment

# file: ch04/mass_assignment.py
class PaymentSchema(MakePaymentSchema):
    id: uuid.UUID
    status: str

@server.post(
    "/payments-vulnerable",
    response_model=PaymentSchema,    #A
    status_code=status.HTTP_201_CREATED,    #B
)
async def make_payment(request: Request):    #C
    with session_maker() as session:
        payload = await request.body()    #D
        payment = PaymentModel(**json.loads(payload))    #E
        session.add(payment)    #F
        session.commit()    #G
        return {    #H
            "id": payment.id,
            "amount": payment.amount,
            "currency": payment.currency,
            "status": payment.status,
        }

To prevent mass assignment, use data validation models to sanitize user input. Create separate models for user input and for output. For example, listing 4.7 creates a MakePaymentSchema model to validate user input. MakePaymentSchema only allows users to specify the amount and currency of the payment, and any additional attributes will be rejected by the server. PaymentSchema represents our full payment model, including its ID and status, and we only use it to validate outgoing data.

Listing 4.7 Constraining user input to prevent mass assignment

class MakePaymentSchema(BaseModel):    #A
    amount: float
    currency: str

class PaymentSchema(MakePaymentSchema):
    id: uuid.UUID
    status: str

@server.post(
    "/payments-safe",
    response_model=PaymentSchema,
    status_code=status.HTTP_201_CREATED,
)
async def make_payment(payment_details: MakePaymentSchema):    #B
    with session_maker() as session:
        payment = PaymentModel(**payment_details.dict())    #C
        session.add(payment)
        session.commit()
        return {
            "id": payment.id,
            "amount": payment.amount,
            "currency": payment.currency,
            "status": payment.status,
        }

Mass assignment is a very widespread vulnerability, and as you can see in the examples above, it poses great threats to our business model and the data privacy of our users. Make sure all your API endpoints and operations have clearly defined data validation models for user input, and ensure those models only allow the presence of legit attributes, such as currency and amount in our payments example.

Mass assignment is one side of broken property level authorization (BOPLA). The other side of BOPLA is Excessive Data Exposure. In the next section, you’ll learn what it looks like and how to prevent it.

4.4.2 Excessive data exposure

Excessive data exposure happens when our API exposes more data than it should. Excessive data exposure comes in different flavors that pave the way for different types of attacks against our APIs. For example, as illustrated in figure 4.8, some APIs expose identifying user information, including their email addresses, dates of birth, and their IDs, through publicly available endpoints. This makes our API vulnerable to a personal identifying information (PII) data breach if threat actors find a predictable way to retrieve user profiles.

Figure 4.8 Excessive data exposure happens when we exposure sensitive data to users who shouldn’t have access to it. For example, when we expose personal user details in a public user search API.

Other applications expose their internal data models directly through the API. A s shown in figure 4.9, tight coupling our data models to our APIs is an anti-pattern, and it’s bad from a design and security perspective. As Mike Amundsen highlights, “your data model is not your object model is not your resource model is not your message model.” (This is known as Amundsen’s Maxim: https://www.amundsens-maxim.com/). From a design perspective, it’s bad because it makes our APIs prone to changes whenever our internal data models change. From a security perspective, it’s bad because it makes our API leak a lot of information about our internal models that threat actors can leverage to abuse and manipulate our system.

Figure 4.9 A common cause of excessive data exposure is tight coupling between our database models and our API data models, which leads to sensitive data leaks if data isn’t thoroughly filtered

How are our internal data models useful to threat actors? Imagine we sell insurance through a website. As with any good insurance business, we ask our customers a series of questions to calculate their risk profile and their premium, and we record that information in our internal data models. As illustrated in figure 4.10, if we leak our internal data models through the API, any user looking directly at the data exchanges between our API and our client (e.g. browser web application) will learn how we model risk and premiums. This information may help threat actors figure out how to abuse our risk model, or how to plan a mass assignment attack that changes their risk profile.

Figure 4.10 Excessive data exposure can reveal sensitive information from our system, like the weights used to calculate insurance premiums. Threat actors can use this information to abuse our business model

Another flavor of excessive data exposure consists of exposing unnecessary objects and identifiers. This typically happens in APIs designed to be consumed by our own clients (i.e. browser or native applications built by our organization). For example, as shown in figure 4.11, an e-commerce application may have special deals for customers with certain loyalty points. Excessive data exposure would happen if our API included the special deals in the response, expecting our client application to filter them out and hide them from unqualified customers.

Figure 4.11 Another flavor of excessive data exposure consists of leaking full objects. For example, when an e-commerce API exposes special deals a customer shouldn’t have access to.

Excessive identifiers are a problem too, as shown in figure 4.12. Imagine we run a ride-sharing business like Uber. Customers can book rides and get the details of their rides through our API. When requesting the details of their ride, they only need to know the model and registration plate of the car that’s going to pick them up, and maybe the first name of the driver too. Customers don’t need to know what the database identifiers of the driver and their vehicle are, and including such information in our API responses would be excessive data exposure. Threat actors could leverage such information to try and get access to try and get ownership of the driver’s account or their vehicle.

Figure 4.12 Another form of excessive data exposure is exposing unnecessary object identifiers. In this example, a ride-sharing application leaks the ID of the driver and their vehicle to the customer.

That’s enough theory. I’m sure you’re wondering, “but, does any of this happen in the real world?” It does, and it’s a major source of security incidents. For example, in January 2022, Twitter (now x.com) fixed a vulnerability that allowed users to submit an email address or phone number through the login flow and discover which accounts they were bound to. In this case, the information about which account were the email address and phone number bound to was completely unnecessary. This vulnerability was due to a code update in June 2021, and sadly, between that date and the fix in January 2022, threat actors managed to harvest a list of user accounts together with the email addresses and phone numbers and put it up for sale (https://privacy.twitter.com/en/blog/2022/an-issue-affecting-some-anonymous-accounts).

More recently, in January 2024, Trello suffered a data breach affecting 15 million users. In this case, a threat actor exploited an unauthenticated endpoint that allowed any user to query the profile of any user on the platform by simply submitting their email address. The data leak included emails, usernames, full names, and other details.[3]

Of course, there are many problems with the Trello breach, including the lack of access controls and the lack of monitoring and rate-limiting. And we must question whether this business flow was necessary at all: should anyone be allowed to query user profiles by email? You can argue that this functionality is useful to find other team members on the platform. For example, if you want to find out what projects or tickets your colleague is working on, it’s useful to be able to look up their profile by email. And that’s where excessive data exposure comes in: when you query the API, you should only get a user profile if you’re connected with them.

Practical example of excessive data exposure

What does excessive data exposure look like in practice? Let’s look at a simplified example that replicates the Trello vulnerability. The code in listing 4.8 implements a GET /query-user endpoint that allows us to query users by email, for example by sending a request like GET /query-user?email=susan@example.com. If the email matches the profile of a user who works on the same project as ours, we return the user profile, otherwise we’ll return a 404 Not Found response. The implementation in listing 4.8 is vulnerable to excessive data exposure, since any matching email returns the corresponding user profile without additional checks, as highlighted in lines 15-17. Check out ch04/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and test the vulnerability.

Listing 4.8 Querying users with excessive data exposure

# file: ch04/excessive_data_exposure.py
class UserClaims(BaseModel):    #A
    sub: str
    project: str

def validate_access():    #B
    return UserClaims(sub=str(uuid.uuid4()), project="The Matrix")

@server.get("/query-user", response_model=UserProfileSchema)
async def query_user_vulnerable(
    email: str, user_claims: UserClaims = Depends(validate_access)
):
    with session_maker() as session:
        requested_user = session.scalar(    #C
            select(UserModel).where(UserModel.email == email)
        )
        if requested_user is None:
            raise HTTPException(
                status_code=404,
                detail=f"User with email {email} not found."
            )
        return {
            "id": requested_user.id,
            "email": requested_user.email,
            "project": requested_user.project,
            "full_name": requested_user.full_name,
        }

To fix excessive data exposure in listing 4.8, we include an additional condition in our database query to check if the user we’re querying for works on the same project as we do. To do this, we first query the database to pull the details of the user sending the request and we keep that information in a variable named requesting_user, as shown in lines 9-11 of listing 4.9. Then, we include an additional condition in the query for the requested user, checking if they work in the same project as the requesting user, as shown in line 14 of listing 4.9. If both users work on the same project, we return the requested user details, and otherwise, we respond with a 404. With the fix in listing 4.7, threat actors won’t be able to scrape our API for random users.

Listing 4.9 Querying users without excessive data exposure

# file: ch04/excessive_data_exposure.py
@server.get(
    "/query-user", response_model=GetUserProfileSchema,
)
async def query_user (
    email: str, user_claims: UserClaims = Depends(validate_access)
):
    with session_maker() as session:
        requesting_user = session.scalar(    #A
            UserModel.id == user_claims.sub,
        )
        requested_user = session.scalar(select(UserModel).where(
            UserModel.email == email,
            UserModel.project == requesting_user.project,    #B
        ))
        if requested_user is None:
            raise HTTPException(
                status_code=404,
                detail=f"User with email {email} not found."
            )
        return {
            "id": requested_user.id,
            "email": requested_user.email,
            "project": requested_user.project,
            "full_name": requested_user.full_name,
        }

This concludes our journey through understanding and tackling broken property level authorization. Now let’s take a look at user groups and roles and how they can make our APIs vulnerable!

4.5 Broken Function Level Authorization (BFLA)

Most applications have a concept of user groups or roles with different permissions and access levels. For example, many APIs distinguish between admin and non-admin users, or between developers and non-developers. Some APIs distinguish more granular types of groups. For instance, a healthcare API may distinguish between nurses, receptionists, and doctors determine the level of access a user has. We call this role-based access controls (RBAC).

DEFINITION

Most applications have a concept of user groups or roles, such as admin users and normal users. Role-based access controls (RBAC) are authorization checks based on such user roles and groups.

Broken Function Level Authorization (BFLA) happens when a user manages to bypass the access constraints of their role. For example, when a non-admin user gets access to admin-only operations. We call this elevation of privilege or privilege escalation. This is also known as vertical privilege escalation, and it’s different from horizontal privilege escalation, where a user can access data from another user with the same role (i.e. BOLA).

DEFINITION

Privilege escalation is an attack strategy in which threat actors attempt to gain access to operations or resources they don’t have access to. For example, when a normal user gets access to an admin endpoint, they’re performing a privilege escalation attack.

The main driver of BFLA is weak RBAC enforcement or lack thereof. As illustrated in figure 4.13, many applications expose admin and/or developer APIs. Typically, those APIs give access to functionality that average users don’t have access to. For example, an admin API may allow viewing and make changes to data from all users. Of course, the idea is that only admin users should have access to the admin API.

Figure 4.13 Most APIs have a concept of user groups or roles with different permissions. In this example, an API has roles for normal users, admins, and developers.

Admin, developer, and similar types of APIs are often deployed on specific subdomains or URL paths, such as admin.example.com or example.com/api/admin. These APIs are sometimes hidden from public API documentation and catalogs to prevent unauthorized users from discovering them. The assumption is, by keeping our admin APIs secret, we won’t attract unwanted threat actors.

As illustrated in figure 4.14, the trouble is these APIs are sometimes built under the assumption that only the right type of users will access them, and hence don’t apply the right access controls on every request. This means that any user who discovers our admin endpoints will instantly elevate their privileges and gain admin access to our API.

Figure 4.14 A common cause of BFLA is lack of checks on whether users are requesting access to the right type of API. In this example, a normal user gets access to an admin API

Do we have real-world examples of BLFA breaches? Sure thing! On Shopify, merchants can give restricted access to their stores to collaborators. Since collaborators have access to such sensitive information, Shopify has robust checks to ensure only the right users become collaborators. However, in September 2017, a cybersecurity researcher discovered a vulnerability in the collaborator onboarding process. After creating two partner accounts with the same business email, the researcher found they could obtain collaborator status with full store access on any account without merchant approval. Thankfully, Shopify fixed the vulnerability within one hour of the report (https://hackerone.com/reports/270981).

Sometimes, BFLA happens without even having to assume another role. For example, in early 2020, cybersecurity researcher Sanjana Sarda found that Bumble, the popular dating website, didn’t enforce restricted access to admin, developer, and premium functionality on its APIs. Due to this vulnerability, Sanjana was able to obtain admin access to the platform, premium features, and to personal information from Bumble’s entire user base.[4]

To protect our APIs against BFLA, we must check that each user is allowed to access the data or functionality they’re requesting according to their user role or group. We do this by implementing role-based access controls (RBAC). How does RBAC work in practice? Different APIs implement RBAC differently, so it depends on your implementation. As shown in figure 4.15, the most common strategy is having a list of user roles, where each role has different access models. For example, an application may have a customer, an admin, and a developer role, and each role may give access to different APIs, endpoints, or datasets. Users typically carry their role in their access token. For example, tokens issued by Auth0 include a custom permissions claim with a list of the user roles and permissions. In this case, our API must validate that the permissions claim contains the right list of values. For example, if a user is accessing our admin API, we must check that the permissions field contains an admin role.

Figure 4.15 A good strategy to prevent BFLA is declaring user roles explicitly in access tokens, and checking those roles on every request across all operations.

How do we mitigate BFLA vulnerabilities in practice? The next section illustrates with a practical example how we check the user’s role by inspecting their access token’s payload, and how we authorize their access against it.

4.5.1 Practical example of preventing BFLA

Imagine we have an admin endpoint like the one in listing 4.10. For simplicity, the endpoint returns a “success!” message if we have access to it. The implementation in 4.10 doesn’t check the user role, so everyone can get access and hence it’s vulnerable to BFLA. Check out ch04/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and test the vulnerability.

Listing 4.10 BFLA-vulnerable admin endpoint

# file: ch04/bfla.py
@server.get("/admin")
def get_admin():
    return "success!"    #A

To prevent BFLA, we must check whether the user has an admin role attached to them. This information is present in the access token, so we must first capture the claims in the token’s payload. Listing 4.11 validates the access token and captures its claims using a validate_token() function using the same approached we followed in listing 4.4. In this case, the user role information is available in the permissions claim, which represents an array of user roles and permissions. To check that the user is an admin, we simply check that the admin role is preset in the permissions array.

Listing 4.11 Mitigating BFLA vulnerabilities

# file: ch04/bfla.py
@server.get("/admin")
def get_admin(user_claims: UserClaims = Depends(validate_token)):
    if "admin" in user_claims.permissions:    #A
        return "success!"    #B
    raise HTTPException(    #C
        status_code=401, detail=f"Unauthorized"
    )

We’ll dive deeper into role-based access controls (RBAC) in chapters 6 and 7, but the example here gives you the gist of it: to mitigate BFLA vulnerabilities, validate the user’s access token and capture its claims, then check the user has the right permissions and roles to access the requested functionality.

The API vulnerabilities we’ve seen so far relate to the ability of a user to access our system or resources within our system. These vulnerabilities are incredibly important, and failing to address them properly exposes our APIs to all sorts of abuses. But they are also relatively easy to address if you understand how the protocols work and how to implement mitigation strategies correctly. In the next section, we introduce a new type of vulnerability that brings more complex yet fascinating challenges on the table: the ability to abuse our business models.

4.6 Unrestricted access to sensitive business flows

Did you try to buy a PlayStation 5 (PS5) online when it launched and found that it was always sold out? Have you tried buying a ticket for a Taylor Swift concert and found they’re all gone? Did you try to book an appointment for your driving test and find all the slots were already booked? All of these are examples of a newly added threat category to the OWASP top 10: unrestricted access to sensitive business flows.

What is a sensitive business flow? Any flow a threat actor can exploit to their benefit and to the detriment of our business. For example, figure 4.16 shows a typical user flow for a referral program: user A sends an invitation to user B to join a ride-sharing platform. When user B joins the platform, user A earns a discount on future rides. If our website doesn’t check the authenticity of user B’s account, then we’re vulnerable to unrestricted access to sensitive business flows.

Figure 4.16 A typical referral program encourages existing users to invite other users to the platform and offer rewards on every successful signup.

Unrestricted access to sensitive business flows exposes our APIs to various threat models, like scalping, review manipulation, scraping, and others. The stories about the sold out PS5s and Taylor Swift concert tickets are examples of scalping. Scalping is the practice of buying out the whole stock of a high-demand product to resell it later at a higher price, and it affects e-commerce stores, booking websites, ticket sellers, and more.

As shown in figure 4.17, scalper bots work by polling your website to check the availability of certain products, tickets, and booking slots. For example, a scalper bot looking to buy and resell a popular brand of shoes may poll your website every few minutes using different IPs. The moment your shoes become available for sale, the bot buys out the whole stock in bulk.

Figure 4.17 Scalper bots poll APIs every few seconds until a product becomes available and then buy the whole stock.

On the surface, scalping may not look like a threat to your business. After all, you’re still selling your products, right? Yes, but it comes at a cost. Scalper bots result in poor customer experience since your products are rarely in stock, and they compromise the performance of your website by constantly polling the API. Besides, scalper bots become the effective seller of your products, which means you don’t own the relationship with your customers anymore. Research by Netacea, a bot detection and prevention technology company, shows that customers are likely to stop buying from your website, post negative reviews, and complain to regulatory bodies when they encounter these issues.[5]

Other examples of unrestricted access to sensitive business flows include manipulation of review and scoring systems, abuse of referral programs, scraping content, manipulation of bidding contests, and more. For example, for a few years now, websites like Amazon and Google are in a constant fight to address the problem of fake reviews. For instance, Amazon sellers buy positive reviews from fake customers and negative reviews for their competitors in an attempt to influence customer behavior.[6] Businesses listed on Google Maps also exploit its review system, and Google has decided to take the matter into legal hands to tackle fake reviewers.[7]

At this point, you may be wondering: but what does all this have to do with technology and APIs? After all, managing inventories, selling products, and handling customer reviews are inherently business functions, aren’t they? Yes, but remember: our job as software developers is to provide business solutions, not just to write code. There’s a lot we can and should do from a technical perspective to address vulnerable business flows.

How do we secure vulnerable business flows using technology? The first step is to mitigate the threat from automated requests. How do we do that? It depends on our user engagement model. As illustrated in figure 4.18, if our customers are meant to access our website through a front-end client, we may want to try and disable access from headless browsers or command-line tools. There’s no foolproof method for disabling headless access to your API, but you can try a few things. Start by checking the request’s user agent. The user agent represents the software used to send the request, and it is usually indicated in the User-Agent request header. Of course, nothing prevents a threat actor from manually setting the value of their User-Agent to a browser, hence this strategy only goes so far. A more effective strategy is serving CAPTCHA challenges to verify there's a human on the other side of the network.

Figure 4.18 If our API is designed for consumption by a browser-based application, a good strategy to mitigate automated attacks is serving CAPTCHA challenges.

Additionally, implement non-human detection patterns. For example, as shown in figure 4.19, if we have an e-commerce website, we may expect users to go through a specific journey to buy our products, like browsing our catalog, checking product descriptions and reviews, adding items to the cart, and finally checking out. The whole process may well take anywhere from a few minutes to a few hours. Flag deviations from this engagement model, throttle requests when you suspect malicious activity, and block users when they’re clearly abusing your API.

Figure 4.19 In a multi-step user flow, we can detect bots by measuring the time intervals between each step. While a legit user may take a few seconds between each step, a script will run through them in miliseconds.

What if we also want to allow users to engage directly with the API? If you want to allow automated access to your sensitive business flows through the API, consider doing it through a partner program and track every API interaction to the last detail. For example, if you run a marketplace for flight tickets and want to open your API to business partners, you want to have very detailed tracking of what each partner is doing with the API and be able to hold them accountable for any manipulation of prices or ticket availability.

You may now be wondering, “but how do we implement these measures in practice?” In the next section, we go through a practical example of blocking scalper bots for our next Taylor Swift concert ticket sale. Let’s get to work!

4.6.1 Practical example of mitigating abuse of vulnerable business flows

Imagine we run a ticketing platform where customers can buy tickets to attend their favorite singers’ concerts. Our API allows users to list the available tickets through a GET /tickets endpoint, and to buy them through a POST /checkout endpoint. Listing 4.12 shows the implementation of those endpoints. To keep the implementation simple, we use an in-memory database represented by two arrays: the tickets_db array represents the list of available tickets, and the recent_transactions array represents a list of recent transactions. For a more realistic example using a SQL database, please refer to the examples in the book’s GitHub repository. Check out ch04/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and test the vulnerability.

Listing 4.12 Ticketing system vulnerable to abuse

# file: ch04/vulnerable_business_flows.py
tickets_db = [...]    Check the GitHub repo for the array contents #A
transactions = []    #B

@server.get("/tickets", response_model=ListTickets)    #C
def list_tickets():
    return {"tickets": tickets_db}

@server.post(
    "/checkout",    #D
    response_model=TicketsPurchase,
    status_code=status.HTTP_201_CREATED,
)
def checkout(
        ticket_details: BuyTickets,
        user_claims: UserClaims = Depends(validate_access),
):
    for ticket in tickets_db:    #E
        if ticket["id"] == ticket_details.ticket:
            recent_transactions.append(    #F
                 Transaction(
                     sub=user_claims.sub, date=datetime.now(timezone.utc)
                 )
            )
            return {
                "id": uuid.uuid4(), "ticket": ticket,
                "amount": ticket_details.amount,
            }
    raise HTTPException(    #G
        status_code=404,
        detail=f"Ticket with ID {ticket_details.ticket} not found."
    )

As it stands, our implementation is vulnerable to scalping. Nothing prevents a user from running a script to buy all the available tickets as soon as they become available. To mitigate this risk, we’re going to implement two measures:

We’ll prevent access from non-browser-based user-agents

We’ll allow users to do only one transaction every 24 hours

The decision to restrict transactions to a 24-hour window may sound unreasonable and hurt real users who may want to acquire more tickets within less than 24 hours for legitimate reasons. A more effective solution would involve a proper threat detection system that can flag suspicious users, and we’ll discuss how to do that in chapters 6 and 11. For this illustration, the simple 24-hour restriction serves our example well.

Regarding the user-agent check, as we discussed earlier, checking the request’s user-agent won’t protect us fully from automated scripts, but it does turn away simple automated scripts that leave the user-agent unmodified. We begin by declaring the list of allowed user-agents, as shown in listing 4.15. A full user-agent header looks like this: “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0.” As you can see, the user-agent contains information about the browser, operating system, computer architecture, and so on. To keep our implementation simple, we’ll simply check that the user-agent header contains a browser.

Listing 4.13 also implements a Transaction model to represent every transaction. Transaction contains a within_24() method that tells us whether the transaction happened within the last 24 hours.

Listing 4.13 Allowed browsers and Transaction model

# file: ch04/vulnerable_business_flows.py
allowed_agents = ["Mozilla", "Google Chrome", "Safari", "Microsoft Edge"]

class Transaction(BaseModel):    #A
    sub: str
    date: datetime

    @property    #B
    def within_24h(self):
        time_diff = (    #C
            datetime.now(timezone.utc).timestamp()
            - self.date.timestamp()
        )
        return (time_diff / 3600) <= 24    #D

The next step is to make use of this functionality to prevent scalper bots. Listing 4.14 implements a prevent_scalping() middleware that brings it all together. First, we check whether the user-agent request header contains a valid browser. If it doesn’t, we respond with a 401 (Unauthorized) status code. Next, we check whether the user is trying to buy a new ticket, and whether they should be allowed. We capture the user identifier from the access token’s payload and check it against our array of recent transactions. If the user made a purchase in the past 24 hours, we respond with a 401 (Unauthorized) status code.

Listing 4.14 Custom middleware to prevent scalping

# file: ch04/vulnerable_business_flows.py
class UserClaims(BaseModel):
    sub: str

def validate_access():    #A
    return UserClaims(
        sub="deb6f47e-fd9d-4b4c-ae4f-c85726ed502c"
    )

@server.middleware("http.request")    #B
async def prevent_scalping(request: Request, call_next):
    if not any(    #C
        agent in request.headers["user-agent"] for agent in allowed_agents
    ):
        return JSONResponse(    #D
            status_code=401, content={"error": "Unauthorized"}
        )

    user_claims = validate_access()    #E

    if request.url.path == "/checkout":    #F
        for transaction in recent_transactions:
            if (    #G
                transaction.sub == user_claims.sub
                and transaction.within_24h
            ):
                return JSONResponse(    #H
                    status_code=401, content={"error": "Unauthorized"}
                )
    return await call_next(request)    #I

This example only scratches the surface of what is possible to prevent threat actors from abusing our user flows, but it gives you an idea of what is possible and how. The best way to protect our APIs against such threats is by understanding the business requirements of our applications and tailoring our mitigation strategies to them.

This concludes our journey through the most common types of API authentication and authorization vulnerabilities. In the next chapter, we look at a different set of vulnerabilities that have to do with API configuration and management.

4.7 Summary

The OWASP top 10 API security vulnerabilities is a list of the most common API threats published by the Open Worldwide Application Security Project (OWASP).

Broken object level authorization (BOLA) occurs when user A accesses private data from user B. To prevent it, enforce robust object-level access controls across all your endpoints, ensuring each request is validated against the user’s permissions.

Broken authentication happens when a threat actor breaks through our authentication system. To prevent it, use Identity as a Service (IDaaS) providers and leverage standard protocols and specifications like Open Authorization (OAuth) and JSON Web Tokens (JWT).

Broken object property level authorization (BOPLA) happens when a threat actor overrides (mass assignment) or accesses (excessive data exposure) a property that should be hidden from them. To prevent it, constrain user input, use strict data models, and make sure you check thoroughly user permissions before determining what data they have access to.

Broken function level authorization (BFLA) occurs when users gain access to the privileges of a user group they don't belong to, for example, when a non-admin user gets access to an admin operation. To prevent it, declare user permissions or roles explicitly in their access tokens and validate their access against all operations in your API.

Unrestricted access to sensitive business flows refers to a threat actor's ability to abuse our business model through our API. A well-known example is scalper bots that buy the whole stock of a product and resell it at a higher price. To prevent it, constrain by design how users engage with your API and counter automated bots by serving CAPTCHA challenges.

[1] Ben Knight, “JSON Web Token Validation Bypass in Auth0 Authentication API,” CyberCS Blog, April 16, 2020 (https://cybercx.co.nz/blog/json-web-token-validation-bypass-in-auth0-authentication-api/).

[2] Egor Homakov documented the exploit in his blog with a post titled “Hacking rails/rails repo,” published on March 4, 2012 (https://homakov.blogspot.com/2012/03/how-to.html).

[3] Alicia Hope, “Massive Trello User Data Leak: Hacker Lists 15 Million Records on a Dark Web Hacking Forum,” CPO Magazine, February 2, 2024 (https://www.cpomagazine.com/cyber-security/massive-trello-user-data-leak-hacker-lists-15-million-records-on-a-dark-web-hacking-forum/).

[4] Becky Bracken, “Dating Site Bumble Leaves Swipes Unsecured for 100M Users,” ThreatPost, November 16, 2020 (https://threatpost.com/dating-site-bumble-swipes-unsecured-100m-users/161276/).

[5] Netacea, How are bots changing buyer behavior?, November 16, 2023, p. 8 (https://netacea.com/research-and-reports/how-are-bot-changing-buyer-behavior/).

[6] Simon Hill, “How to Spot Fake Reviews on Amazon,” Wired, July 8, 2023 (https://www.wired.com/story/how-to-spot-fake-reviews-amazon/).

[7] Renny Hwang, “We’re taking legal action to stop fake review scams,” Google Blog, July 16, 2023 (https://blog.google/outreach-initiatives/public-policy/legal-action-stop-fake-review-scams/).

5 Top API configuration and management vulnerabilities

This chapter covers

Restricting resource consumption

Mitigating Server-Side Request Forgery (SSRF)

Configuring our APIs safely

Managing our API attack surface securely

Consuming APIs safely

We continue our exploration of the most common API security risks by looking at API configuration and management-related categories from the OWASP API top 10. While the vulnerabilities in the previous chapter involved access controls to our system, resources, and business logic flows, the vulnerabilities in this chapter involve abuse of misconfiguration that allow threat actors to trigger random requests from our system, obtain sensitive system information, and more. You’ll also learn about the importance of managing your API attack surface and how threat actors look for old API versions or internal endpoints that are less protected.

As in chapter 4, I illustrate every vulnerability with a coding example whenever that’s possible, and I demonstrate practical strategies to mitigate the risks. I also highlight when a vulnerability is better solved with a different approach, such as using dedicated infrastructure components. If you want to run the code in the GitHub repository for this book (https://github.com/abunuwas/secure-apis/) as you read through this chapter, check out section 4.1 for instructions on how to set up the environment, install the dependencies, and run the vulnerable servers that illustrate each of the vulnerabilities discussed in this chapter. For more details on this section’s code examples, check out the ch05/README.md in this book’s GitHub repository.

5.1 Unrestricted resource consumption

Unrestricted resource consumption is a threat actor’s ability to get unfettered access to our system. The most common type of unrestricted resource consumption attack is denial of service attack (DoS), or more commonly, distributed denial of service attacks (DDoS). As shown in figure 5.1, a DDoS attacks consist on launching multiple concurrent requests against our servers to exhaust all our resources and bring our services down. If our resources scale without limit, as is the case with many serverless services (like AWS Lambda), the likely outcome is pushing our cloud bill beyond our limits, potentially bankrupting our business. The most common way to fend off DDoS attacks is by deploying services that analyze web traffic to detect anomalies, serve CAPTCHA and other challenges to incoming requests, and rate-limit or temporarily ban access when suspicious activity is detected. As you will learn in chapter 9, a popular choice to protect against DDoS attacks is deploying a web application firewall (WAF), and using DoS mitigation technologies such as Cloudflare, Akamai, Radware, Imperva, and others. A layered approach, with multiple levels of protection at different levels of the stack, is likely to work best against this type of attack.

Figure 5.1 Unrestricted resource consumption happens when threat actors get unfettered access to our servers. In this example, a threat actor launches a DDoS attack from three different servers.

A denial of service attack doesn’t always require millions of concurrent requests. If our API exposes a CPU and/or data intensive operation, for example to crunch millions of data points to produce an analytics report or to produce a sales forecast, it may take only a few requests to exhaust our server resources. This has an important consequence: you shouldn’t apply the same rate-limiting rules to all your endpoints. While rate-limiting protects our APIs from abuse, blanket applying the same policy across all endpoints is likely to undermine user experience. For example, if it takes 10 API calls to purchase an item through your e-commerce website and you block users after 8 calls, they’ll eventually stop using your website.

Apply strict rate-limiting rules to your computationally intensive and data-sensitive endpoints to prevent threat actors from abusing them, and maintain a more relaxed policy on your less sensitive endpoints. Align with your stakeholders to understand your API usage patterns and determine what is a reasonable threshold to suspect a DoS attack for any given endpoint or user flow.

Unrestricted resource consumption isn’t just about exhausting your server resources. Consider the case of serverless computing. Serverless offerings, such as AWS lambda, GCP’s cloud run functions, and Azure functions, allow you to run applications without having to worry about provisioning of scaling infrastructure – AWS, GCP, and Azure do it for you. Serverless infrastructure scales as much as needed to meet user demand. In normal situations, this is desirable, but in the middle of a DDoS attack, it can make your cloud costs spiral out of control. This is also known as a cost attack or denial of wallet attack.

DEFINITION

A denial of wallet attack is type of attack in which threat actors exploit the ability of your cloud infrastructure to scale without control to cause financial harm to your organization. This can be accomplished by sending millions of requests from distributed nodes or by triggering operations that are expensive to compute. For more on this types of attacks, see Daniel Kelly, Frank g. Glavin, and Enda Barrett, “Denial of wallet—Defining a looming threat to serverless computing,” Journal of Information Security and Applications, 60 (2021) (https://doi.org/10.1016/j.jisa.2021.102843) and Mark Dorsett, Scott Mann, Jabed Chowdhury, and Abdun Mahmood, “A Comprehensive Review of Denial of Wallet Attacks in Serverless Architectures,” SSRN (July 14, 2025) (http://dx.doi.org/10.2139/ssrn.5350294).

Unrestricted resource consumption vulnerabilities can also be exploited for brute force attacks. As shown in figure 5.2, a common exploit is to engage in brute-force attacks to try and take over other user accounts or gain unauthorized access to resources. How? By sending multiple combinations of username and password to your login endpoint or attempting to forge access tokens.

Figure 5.2 A common exploit of unrestricted resource consumption is conducting brute force attacks to break other user credentials and take over their accounts.

Is this a problem in the real world? You bet it is. DDoS attacks are a daily occurrence. If you have a public API, a DDoS attack is not a matter of if, but when. According to Akamai’s 2024 DDoS: Here to Stay report, DDoS attacks against the financial services industry grew by 154% in 2023, and the trend is likely to continue (https://www.fsisac.com/hubfs/Knowledge/DDoS/FSISAC_DDoS-HereToStay.pdf). Also, according to a study published by the Cyber Express Magazine on December 9, 2024, APIs are more likely to be targets of DDoS attacks, with a 3,000% increase registered in Q3 2024 (https://thecyberexpress.com/api-attacks-surge/).

Sometimes, the very protocols that make the Internet work expose features that threat actors can exploit for DDoS attacks. For example, during the months of August and September 2023, Cloudflare, Google, Microsoft, and Amazon dealt with and mitigated one of the largest DDoS attack registered to that date. The attack was possible due to a vulnerability in the HTTP/2 protocol called “HTTP/2 Rapid Reset.”[1]

Rapid Reset allows attackers to open multiple connections TCP connections with the server by sending a request and canceling it immediately. The vulnerability consists in the server not closing the connection. By canceling the request, the client bypasses the limit of concurrent requests they’re allowed to have with the server. At its peak, Google handled 398 million requests per second.[2] We’re still waiting for a fix to HTTP/2 Rapid Reset, and in the meantime, your best chance to prevent it is by leveraging cloud services like Cloudflare, AWS, GCP, and the likes (see section 9.3 for more details on HTTP/2 Rapid Reset).

What about brute force attacks? In February 2016, AppSecure discovered that Facebook’s beta subdomain (beta.facebook.com) didn’t implement rate-limiting policies, allowing them to brute force Facebook’s password reset flow. When resetting their password, users would have to enter their phone number or email address, and Facebook would send a 6-digit confirmation code to the number or address indicated. Since Facebook’s beta subdomain didn’t implement rate-limiting, AppSecure was able to crack the 6-digit code and reset the password.[3]

In October 2023, DNA-testing company 23andMe reported that nearly 7 million user accounts were compromised due to a brute force attack against their website (https://techcrunch.com/2023/12/04/23andme-confirms-hackers-stole-ancestry-data-on-6-9-million-users/). And in early 2025, a threat actors launched a large-scale brute force attack using up to 2.8 million IP addresses to try and get access to networking devices manufactured by Palo Alto Networks, Ivanti, and SonicWall (https://www.bleepingcomputer.com/news/security/massive-brute-force-attack-uses-28-million-ips-to-target-vpn-devices/). If you browse HackerOne, you’ll find similar examples for dating application Bumble (https://hackerone.com/reports/743545), Twitter (https://hackerone.com/reports/819930), South African Telecommunications provider MTN Group (https://hackerone.com/reports/1060541), and others. You’ll find that hackers use different strategies, like changing their IP while cracking the same account, waiting an hour between batches of requests, and so on.

To protect yourself from unrestricted resource consumption threats, implement rate-limiting policies, and remember to do it on a per-operation and user flow basis. It’s worth tracking both the source IP of the attacks and the accounts being targeted, as well as session IDs and device fingerprints. For example, after 10 or 15 attempts to log into a specific account, lock access to the account and let the account owner know how to safely reset their password.

Encourage your users to enable multi-factor authentication and serve CAPTCHA challenges on sensitive endpoints. Monitor all traffic all the time to detect suspicious activity and react accordingly. Configure spending limits and create alerts to be sent when your spending crosses a certain threshold. Cloud services like Cloudflare, AWS, GCP, Azure, and others usually have built-in protection against many forms of DDoS and brute force attacks. The following section illustrates a code solution to tracking user access to sensitive endpoints.

5.1.1 Addressing unrestricted resource consumption with code

Imagine we have a login endpoint like the one shown in listing 5.1. Without additional checks and measures, listing 5.1 is vulnerable to brute force attacks. Threat actors can take over any user account by attempting multiple combinations of username and password. Check out ch05/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and test the vulnerability.

Listing 5.1 Login endpoint vulnerable to brute force attacks

# file: ch05/unrestricted_resource_consumption.py
@server.post("/login", response_model=LoginSuccessSchema)
def login(login_details: LoginSchema):
    for user in users_db:
        if (    #A
            user["email"] == login_details.email
            and user["password"] == login_details.password
        ):
            return {"message": "success!"}    #B
    raise HTTPException(    #C
        status_code=401, detail={"error": "Wrong email or password"}
    )

How do we prevent brute force attacks against the login endpoint in listing 5.1? We want to limit the number of login attempts to any user account. For the purpose of this example, let’s say that if an account has been the target of 10 login attempts within 24 hours, we’ll lock the account and prevent any further logins. Equally, we want to track who’s trying to break into our system, and so we’ll monitor the origin IP of the login requests. If an IP sends 10 login requests to our system within 24 hours, we’ll block it. Bear in mind that IPs are considered personally identifiable data (PII) under certain legislations such as Europe’s General Data Protection Regulation (GDPR), so you’ll have to let your users know and consent to this. Listing 5.2 implements a prevent_brute_force_login() middleware that implements these measures.

To keep it simple, listing 5.2 keeps track of login activity with a few in-memory objects:

login_attempts_ips is a dictionary that keeps track of the number of login attempts per IP.

login_attempts_accounts is a dictionary that keeps track of the number of login attempts per account.

blocked_ips is a list that contains the blocked Ips.

locked_accounts is a list that contains the blocked accounts.

In a production application, you’ll use a cache store like Redis instead of in-memory objects. The first thing we check in listing 5.2’s (line 10) middleware is whether the originating request’s IP is blocked. If it is, in line 11 we respond with a 401 response and an error message indicating that the IP is temporarily blocked. Next, in line 13, we check if the request is targeting the /login endpoint. If it’s not, in line 14 we move on with a call to the next middleware to finish processing the request. After that, in line 16 we retrieve the JSON payload from the request and check if the email account is locked (line 18), in which case we return a 401 response with a helpful error message (line 19). If neither the IP is blocked nor the account locked, we increase the activity counters and continue processing the request.

Listing 5.2 Preventing brute force attacks

# file: ch05/unrestricted_resource_consumption.py
login_attempts_ips = defaultdict(int)    #A
login_attempts_accounts = defaultdict(int)

blocked_ips = []    #B
locked_accounts = []

@server.middleware("http.request")    #C
async def prevent_brute_force_login(request: Request, call_next):
    if request.client.host in blocked_ips:    #D
        return JSONResponse(status_code=401, content={"error": "Blocked"})

    if request.url.path != "/login":    #E
        return await call_next(request)

    payload = await request.json()    #F

    if payload["email"] in locked_accounts:    #G
        return JSONResponse(status_code=401, content={"error": "Locked"})

    login_attempts_ips[request.client.host] += 1    #H
    login_attempts_accounts[payload["email"]] += 1

    if login_attempts_ips[request.client.host] == 10:    #I
        blocked_ips.append(request.client.host)

    if login_attempts_accounts[payload["email"]] == 10:    #J
        locked_accounts.append(payload["email"])
        login_attempts_accounts[payload["email"]] = 0

    return await call_next(request)    #K

Listing 5.2 implements simple but effective checks. You can apply these checks in your web middleware, or you can move them to your proxy, API Gateway, or any other front-facing technology so long as they support custom functionality like this. Is 10 attempts the right number? It totally depends on your use cases. If your application encourages users to log in often, use a higher counter.

You can also associate IPs with accounts. For example, if sarah.connor@example.com typically logs in from a certain IP, you can allow more login attempts from that account, and when she logs in from a new IP, send an email to confirm the new IP is legit. If someone tries to log in from an IP that’s not currently bound to any account, you want to pay extra attention to that IP and maybe allow less attempts. You can also track device fingerprints and apply velocity checks for factors such as geolocation.

DEFINITION

A velocity check analyzes the speed and frequency of operations, accounting for relevant factors such as IP, geolocation, device fingerprint, user account, or resources involved. This type of analysis is common in sectors vulnerable to fraud, such as banking and payments. For example, when a credit card is stolen, criminals will try to use its funds in a short period of time before the card gets blocked. To counteract these actions, a bank or a payments processor applies velocity checks to analyze the number of times a payment was attempted with the same credit card in a given time frame, and they analyze the data by IP, geolocation, device fingerprint, and other factors, to determine if the operations are fraudulent. To learn more about velocity checks, check out this excellent post by Stripe: https://stripe.com/gb/resources/more/what-is-a-velocity-check-in-payments-what-businesses-should-know.

You can apply similar checks to other endpoints, like data-sensitive or CPU-intensive endpoints to prevent resource exhaustion and data leaks. And remember, your constraints should reflect your user engagement model. If a user must go through 15 API requests to accomplish a goal, then you shouldn’t block them after 15 calls. In the next section, we tackle an even more complex problem: protecting our APIs against server-side request forgery.

5.2 Server-Side Request Forgery (SSRF)

Server-side request forgery (SSRF) is the ability to trigger malicious requests to internal and external services from our servers. The typical attack vector for SSRF is user input in the form of URLs that our server must call. Why would we want to allow users to trigger external HTTP calls from our servers? There are a few good use cases for this, like allowing users to specify their avatar URL and configuring webhooks.

How do SSRF attacks occur? Imagine we run a payments API like Stripe. Like Stripe, our API allows you to configure webhooks to receive notifications when certain events happen. For example, when a payment succeeds, a subscription is created, or when a payment fails. As illustrated in figure 5.3, this allows you to create automated responses to such events to trigger follow-up actions or handle errors.

Figure 5.3 Webhooks represent actions, typically API calls, that users can configure on certain events. In this example, a payments API offers the possibility to trigger webhooks when a payment is successfully processed.

Since webhooks allow users to include custom URLs, threat actors may be able to configure them to trigger malicious requests from our system. As shown in figure 5.4, threat actors can configure webhooks to scan our local network in search for specific databases or other services. The responses to those requests may reveal important information about our internal system configuration and architecture. For example, it may reveal that we run a PostgreSQL database or a Jenkins server.

Figure 5.4 Webhooks are one of the most common vectors of SSRF attacks. In this example, a threat actor configures a webhook to run a local network scan.

How is information about our internal services useful to threat actors? It allows them to refine and prepare their attack strategies. For example, knowing that we run a PostgreSQL database allows them to focus their efforts on SQL injection attacks that target PostgreSQL-specific features and vulnerabilities. If they discover a Jenkins server, they may be able to hijack it to launch their own processes, like a DDoS attack against other websites.

Are attacks like this realistic? Yes, they are. For example, in September 2022, Cider Security, a cybersecurity company, discovered a vulnerability in GitLab webhooks that allowed them to hijack Jenkins servers from other organizations, download malicious programs and execute them. If you run a Jenkins or other custom continuous integration (CI) server in your own infrastructure and it’s connected to GitLab via webhooks, make sure you only allow incoming traffic from restricted IPs.[4]

Another common exploit in SSRF vulnerabilities is triggering calls to endpoints that are known to reveal sensitive information and credentials. For example, EC2 instances on AWS are known to expose credentials through their metadata service, which is available locally on the http://169.255.169.254 address. A threat actor with access to this endpoint via SSRF can easily query the IAM roles associated with the EC2 instance and pull their credentials. Those credentials can later be used to make API calls directly to AWS.[5]

In 2019, Capital One suffered one of the largest data breaches in recent years due to an SSRF vulnerability involving EC2’s metadata service. As shown in figure 5.5, Capital One’s SSRF vulnerability was due to a misconfigured Web Application Firewall (WAF) that could be tricked into relaying requests to internal services, such as EC2’s metadata service. The exploit allowed Paige A. Thompson, a former AWS employee, to gain access to Capital One’s AWS account and steal records from over 100 million credit applications, including personal details such as names, postal addresses, phone numbers, dates of birth, income, and more.[6]

Figure 5.5 Capital One’s SSRF attack happened due to a misconfiguration in their WAF that allowed the threat actor to relay a request to their EC2 instance metadata endpoint and retrieve their AWS access keys.

In response to Capital One’s breach, AWS released a new version of EC2’s metadata service (IMDSv2) in November 2019. IMDSv2 requires calls to EC2’s metadata service to include a temporary token to authorize the requests. This prevents services from accessing the metadata service with a single GET request. Since March 2023, IMDSv2 ships by default on all Amazon Linux instances. If you have any EC2 instances running in your AWS account, make sure you upgrade them to IMDSv2.

How do we protect our APIs from SSRF? I have some good news and bad news. The good news is that we can take a lot of steps to mitigate the risks associated with SSRF. The bad news is, there’s no way to fully prevent SSRF. What measures you can take depends on your use cases. For example, if you want to allow users to provide their avatar URL, include a whitelist of acceptable domains from which the avatar can be retrieved.

What if you need to allow users to submit completely random URLs, for example for webhooks? At the very least, prevent them from making calls to your localhost and other sensitive endpoints, such as AWS EC2’s metadata service and your local network. Call user-provided URLs from isolated instances without access to resources within your internal network, for example without access to your database and other services. If possible, use an outbound request proxy to inspect the API call and evaluate its risk profile. Lunar.dev (https://www.lunar.dev/) offers one of the best products in this space, and Google Cloud has a similar offering too (https://cloud.google.com/blog/products/identity-security/introducing-secure-web-proxy-for-egress-traffic-protection). In the next section, we walk through a practical example of implementing some of these measures.

5.2.1 Practical example of mitigating SSRF

A common SSRF exploit is scanning our local network to discover which services and applications are accessible. Listing 5.3 simulates an API running together with internal services. The listing implements two endpoints: an internal secret GET /secret endpoint that returns sensitive internal data, and a GET /ssrf endpoint that allows us to submit a URL, call the URL, and returns its output. Since the GET /secret endpoint is internal, we hide it from the API documentation by setting the include_in_schema parameter to False. Check out ch05/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and test the vulnerability.

Listing 5.3 API vulnerable to SSRF

# file: ch05/ssrf.py
@server.get("/secret", include_in_schema=False)    #A
def secret():
    return "secret"

@server.get("/ssrf ")
def ssrf (url: str):
    response = requests.get(url)    #B
    return response.content

The GET /ssrf endpoint doesn’t constrain what URLs we can call and hence is vulnerable to SSRF. For example, if we ask the endpoint to call the following URL: http://localhost:8000/secret, it calls our internal secrets endpoint and leaks the sensitive information.

DEFINITION

A denylist (also called blacklist or blocklist) is a list of URLs, IPs, and domain names whose access is denied. We can use denylists to define origins that are forbidden from accessing our services, or to define targets that our services and/or users are forbidden from accessing. The opposite of a denylist is an allowlist (also called whitelist), which defines URLs, IPs, and domain names whose access is allowed. Denylists and allowlists are fundamental access control strategies in networking and are extensively used in firewalls and other security components of a system.

To prevent threat actors from calling our internal secrets service via SSRF attacks, we constrain what URLs users are allowed to submit by creating a denylist, which is a list of forbidden domains. For example, in listing 5.4, we define a denylist that contains the domains we want to prevent threat actors from calling, which in this case are localhost and 127.0.0.1. If a user submits a URL that contains one of those domains, the request will be rejected with a 422 response. Depending on your network topology, you may want to include additional IPs, IP ranges, and domains in your denylist.

Listing 5.4 Mitigating the risk of SSRF attacks to the localhost

# file: ch05/ssrf.py
denylist = ["localhost", "127.0.0.1"]    #A

@server.get("/ssrf")
def ssrf (url: str):
    if any(domain in url for domain in denylist):    #B
        raise HTTPException(
            status_code=422,
            detail=f"Forbidden domain in {url}"
        )
    response = requests.get(url)    #C
    return response.content

As I explained earlier, in many cases it’s very difficult to implement full protection against SSRF vulnerabilities, but the measures I’ve shown you in this section go a long way to mitigating those risks. In the next section, we tackle another common source of vulnerabilities: security misconfiguration.

5.3 Security Misconfiguration

Security misconfiguration refers to the incorrect configuration of our services, leaving open holes in the system that can expose internal and sensitive information. It includes anything from unauthenticated Jenkins servers to publicly accessible S3 buckets, leaking stack traces through error messages, deploying an internal API to a public network, exposing sensitive data and credentials in client code, or forgetting to enable authorization checks on each API call.

Security misconfigurations make our applications vulnerable to multiple types of attacks and exploits. Let’s see some examples. As illustrated in figure 5.6, running our server in debug mode often makes our application leak stack traces and other internal details in error messages, hence revealing sensitive information about the internal implementation details of our system. Threat actors can leverage this information to design and plan their attack strategies.

Figure 5.6 A common consequence of security misconfiguration is leaking stack trace details in error responses. Stack traces often reveal sensitive information about our internal system configuration, and threat actors can exploit this information to design better strategies to break into our system.

Internal APIs typically expose highly sensitive data and operations. As we saw in section 3.4, we should treat internal APIs as if they were public. However, in practice, internal APIs are often less protected than public APIs and sometimes have no authorization at all. Hence, a simple configuration mistake that exposes our internal APIs to a public network makes our organization highly vulnerable to a data breach. Finally, leaking API keys and other secrets in our source code, for example through frontend applications, gives threat actors unchecked access to sensitive APIs in our system.

Security misconfigurations are widespread, and they represent a leading cause of API breaches. For example, in November 2022, Eaton Zvere, a cybersecurity researcher, discovered a vulnerability in Toyota’s Global Supplier Preparation Information Management System (GSPIMS) that allowed him to gain access to confidential data related to Toyota and its suppliers.

Zvere discovered an endpoint in Toyota’s backend API that allowed him to generate an access token for any email, provided it was a valid corporate email from a Toyota employee, without any further checks. Zvere also discovered an endpoint that returned employee information given their email, which he used to find users with admin access. Putting both findings together, Zvere was able to gain admin access to the whole platform.[7]

Toyota’s GSPIMS application probably brought together most of the OWASP top 10 vulnerabilities, but how exactly does it qualify as a security misconfiguration? GSPIMS was meant for internal access to Toyota employees and its suppliers. The fact that the application was accessible to anyone on the Internet meant it was deployed with the wrong network configuration, and hence it qualifies for a security misconfiguration breach.

How do we prevent security misconfiguration exploits? It all begins with secure coding and deployment practices. For example, leverage infrastructure as code (IaS) to apply secure configuration consistently across all your computing resources, and implement an automated and repeatable deployment pipeline that prevents random manual changes. Create a smoke test suite that runs against every deployment and checks there’s no misconfiguration. For example, checking there are no stack traces in error messages and that CORS policies are correctly enforced. In the next section, we analyze a practical example of an API vulnerable to security misconfiguration exploits and go through the process of fixing it.

5.3.1 Practical example of mitigating security misconfiguration

Imagine we have an API endpoint to list users like the one in listing 5.5. The endpoint is GET /users and it has an optional query parameter order_by. order_by allows us to sort users by a given attribute, and by default it sorts them by email. To run this example, please refer to the GitHub repository for this book, as there are database models and fixtures needed to make it work.

As you can see in the second line in listing 5.5, we are running the server in debug mode. This makes our API vulnerable to security misconfiguration exploits. If you run a simple cURL request against the endpoint, like curl http://localhost:8000/users, you get a list of users. Now, if you try to order the results by a non-existing column, like asdf, the API will raise an error, and since we’re running in debug mode, you’ll get a full stack trace of the error. To reproduce this error, run the following request: curl http://localhost:8000/users?order_by=asdf. Check out ch05/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and test the vulnerability.

Listing 5.5 API vulnerable to security misconfiguration exploits

# file: ch05/security_misconfiguration.py
server = FastAPI(debug=True)    #A

@server.get("/users", response_model=ListUsersSchema)
def list_users(order_by: Optional[str] = "email"):    #B
    with session_maker() as session:
        users = list(    #C
            session.scalars(select(UserModel).order_by(text(order_by)))
        )
        return {"users": users}

The stack traces contain information about our file paths, runtime, libraries, database system, and code context of the error. It also leaks the full database query used to query the records. Threat actors can leverage this information to tailor their attack strategy against our system. For example, by knowing that we use a specific type of SQL database, they can narrow down their injection strategy to statements applicable to our database system. They can also exploit this vulnerability to discover hidden properties of our database models by assigning different values to the order_by parameter.

There are many things we can do to improve the example in listing 5.5, like restricting the values of order_by with an enumeration and checking that the fields exist in our models before running the query. From a security misconfiguration perspective, all we need to do in this case is set the debug flag to False: server = FastAPI(debug=False). Thankfully, FastAPI sets debug to False by default, but some other frameworks don’t, so make sure those flags are set to their correct values across all your environments.

Now that we know about the perils of security misconfigurations, it’s time to venture into the lands of API inventory management. What does that even mean, I hear you ask? Let’s discover it in the next section!

5.4 Improper Inventory Management

Improper inventory management refers to insufficient or absent management of our API versions, endpoints, environments, and documentation. Insufficient management of those assets leaves open holes in our system, contributes to API sprawling, and makes it difficult to understand our attack surface. How is this a problem? Let’s break it down.

Most APIs change over time, and often that means we need to release new versions and deprecate the old ones. If our API has external consumers, we typically follow a structured process. Let’s say we currently run version 1 (v1) of our API and we are going to release version 2 (v2). As shown in figure 5.7, the first step is to release v2, deprecate v1 and announce when it’ll be retired, and finally, retire v1.

Figure 5.7 To upgrade our API version, we first release the new version and set the sunset deprecation header in the old version. Finally, when the date is due, we retire the old version.

The standard way to deprecate an API is using the Deprecation and Sunset headers. Deprecation is a timestamp indicating when the API was or will become deprecated, while Sunset is the date when the API will become unavailable. Optionally, we include a Link header to point our consumers to our deprecation policy.[8] For example, the below headers indicate that our API will be deprecated on March 27, 2027, and will be retired on June 30, 2027:

Deprecation: @1806129442
Link: <https://example.com/deprecation>; rel="deprecation"; type="text/html">
Sunset: Wed, 30 Jun 2027 23:59:59 GMT

On June 30, 2027, we retire the API, i.e. we deprovision it and ensure not available anymore. The process to release new API versions and retire old ones is well-understood, so what’s the problem? The problem is, often we simply forget to deprovision and retire the old versions. In particular, when shutting down a previous version is a manual process, the task can easily slip through the cracks. And without proper API version management tools like API Gateways, the old version may get accidentally redeployed and exposed again. This is how we end up with zombie APIs, which are a major concern in API security, as we discussed in section 3.5.

Improper inventory management also opens the door for shadow APIs. As we learned in section 3.5, shadow APIs are APIs released without oversight, often with less security controls and improper monitoring and visibility. The trouble is, shadow and zombie APIs are often less secure - they may include insufficiently protected endpoints and expose sensitive data and operations and tend to be improperly monitored. If a threat actor discovers them, they may be able to break into our system and steal our data without us noticing.

Shopify offers an example of the risk of leaving old APIs around. Shopify’s infrastructure runs on Google Cloud (GCP), and similar to AWS EC2’s metadata service, GCP’s virtual machines run a metadata service that allows us to access sensitive information and credentials. In April 2018, André Baptista, co-founder of cybersecurity company Ethiack, discovered that Shopify’s virtual machines were still using a deprecated version of GCP’s metadata service, which was vulnerable to SSRF attacks. By exploiting this vulnerability, Baptista managed to download Shopify’s SSH keys and connect to their Kubernetes control plane (https://hackerone.com/reports/341876).

What about API sprawling? As explained in section 3.5 and illustrated in figure 5.8, API sprawling refers to the proliferation of APIs out of control, often duplicating functionality, bypassing security checks, and without documentation. The problem with API sprawling is we lose control of our attack surface. If we don’t know what APIs we’re exposing, we won’t be able to protect and monitor them adequately. This is great news for your hackers and very bad news for your organization. Unsurprisingly, according to Traceable’s 2023 State of API Security report, 48% of organizations list API sprawl as their top security challenge.[9]

Figure 5.8 Ill-maintained API documentation is a common source of improper asset management exploits. In this example, a threat actor discover undocumented and unprotected sensitive endpoints.

To protect your APIs from improper inventory management vulnerabilities, you must take control of your attack surface. Automate as many processes as you can. For example, when you decide you’re going to deprecate an API version, automate its retirement so no one has to do it manually. Leverage API gateways and web application firewall technologies to restrict what API versions and operations can be exposed. Use dynamic surface attack discovery technologies that monitor traffic to your network and discover undocumented endpoints. We’ll see practical examples of how to use those technologies in chapter 8.

So far, we’ve focused on vulnerabilities related to APIs we build and expose. But what about the APIs we consume? Should we just trust blindly the APIs we consume from third parties? In the next section, we analyze this question at length!

5.5 Unsafe Consumption of APIs

We all consume APIs, whether it’s internal or third-party APIs. We use internal APIs to integrate with other services in our platform, typically in the context of a microservices or distributed architecture. And we use third-party APIs to incorporate functionality that would be very difficult or too costly to build from scratch, such as sending emails, managing calendars, geolocation, mapping, processing payments, and so on.

For years, we’ve consumed external APIs and didn’t question the risks involved. But hackers are catching up – they always do. It’s perfectly possible to put our services at risk through integrations with third-party APIs. For example, imagine we use an external service to pull user details such as their postal address. As shown in figure 5.9, a threat actor could include a SQL injection statement as their address, and when our application fetches their details, we’ll be exposed to a SQL injection attack.

Figure 5.9 Lack of validation and sanitization on the data coming from third-party APIs exposes our applications to unsafe consumption of APIs exploits. In this example, a threat actor inserts a SQL injection statement in a third-party API. When our API consumes that data, it runs against our database with devastating consequences.

Similarly, if our application queries data from other APIs to display directly on our user interface, we could get exposed to cross-site scripting (XSS) and similar attacks. For example, imagine we run an application that allows business analysists to search for companies in specific sectors. To aggregate data, our application pulls company information from various business directories. If a company owner names their company with a malicious XSS tag, such as ><script src=https://examples.com/malicious-script.js>Ltd, and that name gets dynamically inserted in our HTML without previous filtering and sanitization, we have the perfect recipe for a data breach. For example, a threat actor could use this attack vector to insert a script that sends user credentials to their own server.

Sounds unrealistic? You’d be surprised. In October 2020, a British software developer registered a company under the name of ><SCRIPT SRC=HTTPS://MJT.XSS.HT> LTD. Within days of the registration, Companies House, UK’s companies register, determined that the newly incorporated company’s name could pose a threat to a small number of its customers, and hence forced the company to change its name. The company is still active under the name of “That company whose name used to contain HTML script tags Ltd.”[10]

TIP

A useful strategy to mitigate the risk of XSS and other exploits such as clickjacking is using a robust content security policy (CSP). You define your website’s CSP in your Content-Security-Policy header, and it allows you restrict the execution of malicious scripts from other websites. The CSP header must be included in your website’s HTML page to instruct the browser how to load scripts. For an in-depth analysis of CSP and how it helps prevent XSS, check out Philippe de Dyck’s excellent article “Defending against XSS with CSP”, Auth0 Blog (April 11, 2022) (https://auth0.com/blog/defending-against-xss-with-csp/).

SQL injection and XSS are some of the most serious vulnerabilities we may get exposed to when we consume external APIs, but they’re not the only ones. When our applications depend on third-party APIs, errors in those APIs can cascade into our servers and cause unexpected behaviors. If the APIs we consume become unavailable or start returning invalid or malformed payloads, our services may throw unexpected errors or end up persisting corrupted data in our databases. To prevent situations like that, always make sure you validate and sanitize output from external APIs. Make sure you can handle malfunctioning external APIs, for example, when they fail to respond to a request or encounter a server error, by implementing the circuit breaker pattern and graceful degradation, which allows your applications to continue operating in the face of unexpected system failures. Outbound proxies like Lunar.dev (https://lunar.dev) help you handle external API failures.

DEFINITION

Circuit breaker is a commonly used pattern in distributed architectures that helps us prevent cascading failures when a dependency in our system fails to behave as expected. Such dependencies can exist within our system (e.g. an internal service) or outside our system (a third-party API integration). The idea is to use an outbound proxy to monitor the health of the external dependency and automatically return a useful error if the dependency doesn’t respond on time or at all, or behaves in unexpected ways.

Finally, with the rise of large language models (LLMs) as a service like ChatGPT, we’re seeing a growing number of applications that offer AI services by building wrappers around LLMs.

To mitigate the risk of unsafe consumption of APIs, always validate and sanitize the data coming from other APIs. For example, if the API is documented with an OpenAPI specification, create your own validation models based on the response schemas, and validate the API responses against those schemas at runtime using robust libraries for data validation. If the responses are invalid, reject them and contact the API provider about this. Make sure you carefully parametrize all database queries involving data coming from outside your system to mitigate the risk of SQL injection attacks via poisoned payloads. And finally, ensure you can handle malfunctioning external APIs when they fail to behave as expected to ensure such events don’t cause cascading failures in your system.

In the next section, we walk through a practical example of securing our own servers when consuming external APIs.

5.5.1 Addressing unsafe consumption of APIs in practice

To illustrate unsafe consumption of APIs, listing 5.6 implements two endpoints that represent two different APIs (server-a and server-b):

GET /server-a/unsafe-api represents an unsafe third-party API

GET /server-b/unsafe-consumption represents an API vulnerable to injection from third-party APIs

I’m adding both endpoints within the same API to keep the example simple, but imagine they run on different servers. GET /server-b/unsafe-consumption represents a characters API. If you check the GitHub repository for the book, you’ll see database fixtures for various characters from the Harry Potter universe. Each character has a status such as muggle, wizard, and witch. To know the status of each character, we call an external API, here represented by GET /server-a/unsafe-api. In listing 5.6, the example is hardcoded to update Harry Potter’s status. Sadly, GET /server-a/unsafe-api contains unsafe data and it returns a SQL injection statement. If you call GET /server-b/unsafe-consumption, you’ll see how the status of all characters is set to an empty string. This is an example of unsafe consumption of APIs: we’re not validating the external API’s output and we’re not handling it safely. Check out ch05/README.md in this book’s GitHub repository for instructions on how to run the vulnerable server for this listing and test the vulnerability.

Listing 5.6 API vulnerable to injection from third party APIs

# file: ch05/unsafe_consumption_apis.py
server = FastAPI()

@server.get("/server-a/unsafe-api")
def unsafe_api():
    return {
        "status": "';--"    #A
    }

@server.get("/server-b/unsafe-consumption")
def unsafe_consumption():
    status = requests.get(
        "http://localhost:8000/unsafe-api"    #B
    ).json()["status"]
    with session_maker() as session:
        session.execute(text(    #C
            f"update character set status = '{status}' "
            f"where name = 'Harry Potter'"
        ))
        session.commit()

How do we protect our API from unsafe consumption of data from third-party APIs? A good strategy is parametrizing all database queries, for example by using SQLAlchemy’s query API instead of running a raw SQL statement. If we do that, we’ll prevent the bulk update we saw earlier, but we’ll have a problem: Harry Potter’s status will be updated as a SQL injection statement. That’s not good. Harry Potter is a wizard, and his status shouldn’t be updated if the external API doesn’t return valid data.

To sanitize data from the external API, we create our own validation model. Listing 5.7 implements a Pydantic model that validates the external API’s responses. We know the character’s status can only have one of three values, so we create an enumeration to validate that. If validation fails, our server will throw a 500 (Internal Server Error) response and no record will be updated. We are letting our data models raise a validation error when the data coming from the external API is invalid, which results in the 500 (Internal Server Error) status code response from the API. Alternatively, we could wrap the validation step within a try/except block and return a different error, such as a 400 (Bad Request) status code. Whichever option you choose, make sure the error makes its way to the logs so you can visibility of it, and that the API consumer can process and understand the error message. Logging this type of error not only gives you visibility of the problem, allowing you to trace and debug it, but it also paves the way for early detection of potential attacks or third-party issues, as you’ll learn in chapter 11.

Listing 5.7 Preventing injection from third party APIs

# file: ch05/unsafe_consumption_apis.py
class StatusEnum(str, enum.Enum):    #A
    wizard = "wizard"
    witch = "witch"
    muggle = "muggle"

class CharacterStatus(BaseModel):    #B
    status: StatusEnum

@server.get("/server-b/safe-consumption")
def safe_consumption():
    status = requests.get("http://localhost:8000/unsafe-api").json()    #C
    status_validated = CharacterStatus(status=status)    #D
    with session_maker() as session:
        session.execute(text(
            f"update character set status = '{status_validated.status}' "
            f"where name = 'Harry Potter'"
        ))
        session.commit()

APIs are pervasive in modern applications, and we all consume APIs to use services we don’t have the resources to build ourselves, like emailing and geolocation. However, all APIs are prone to vulnerabilities and exploits, including external APIs, so it’s important to put measures in place to protect our applications from third-party API exploits. The strategies we’ve discussed in this section will help you accomplish that.

This concludes our journey through the main types of vulnerabilities in APIs. We’ve seen tons of real-world examples and discussed plenty of practical examples. There’s a lot to take here, so take your time to digest the content, and make sure you check out the book’s GitHub repository to run the full examples and understand how they work. In the next chapter, we’ll dive deeper into the intricacies of API security by analyzing various categories of design vulnerabilities and how to tackle them.

5.6 Summary

Unrestricted resource consumption happens when threat actors gain unfettered access to our APIs, for example to launch a DDoS or a brute force attack. To prevent it, monitor and track all requests and throttle or ban consumers that show unusual patterns.

Server-side request forgery (SSRF) is a threat actor's ability to trigger malicious requests from our servers. To prevent it, create allowlists with the URLs, IPs, and domains your users are allowed to submit to your APIs, and denylists to forbid access to sensitive or malicious domains and endpoints.

Security misconfiguration refers to the incorrect configuration of our services leading to vulnerabilities. Prevent it by automating deployments and configuration management, and by running smoke tests to detect common misconfigurations.

Improper inventory management refers to inadequate or absent management of our API versions, endpoints, environments, and documentation. To prevent it, use proper API versioning and deprecation strategies, configure your firewalls to restrict traffic to your documented endpoints, and continuously monitor all traffic to discover undocumented attack surface.

Unsafe consumption of APIs occurs when threat actors manage to cause disruption to our services by poisoning data in third-party APIs we consume. To prevent it, validate and sanitize all output from third-party APIs.

[1] Wes Davis, “Cloudflare, Google, and Amazon explain what’s behind the largest DDoS attacks ever,” The Verge, October 10, 2023 (https://www.theverge.com/2023/10/10/23911186/ddos-http2-vulnerability-blocked-amazon-aws-cloudflare-google-cloud).

[2] See the excellent analysis of this vulnerability by Juho Snellman and Daniele Iamartino, “How it works: The novel HTTP/2 ‘Rapid Reset’ DDoS attack,” Google Cloud Blog, October 10, 2023 (https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack).

[3] AppSecure, “I figured out a way to hack any of Facebook’s 2 billion accounts, and they paid me a $15,000 bounty…” FreeCodeCamp Blog, February 9, 2018 (https://www.freecodecamp.org/news/responsible-disclosure-how-i-could-have-hacked-all-facebook-accounts-f47c0252ae4d/).

[4] Omer Gil and Asi Greenholts, “Abusing Repository Webhooks to Access Internal CI Systems at Scale,” Palo Alto Networks Blog, September 13, 2023 (https://www.paloaltonetworks.com/blog/prisma-cloud/repository-webhook-abuse-access-ci-cd-systems-at-scale/)

[5] See the detailed guide on hacking EC2’s metadata service by Nick Frichette, “Steal EC2 Metadata Credentials via SSRF,” Hacking The Cloud, August 1, 2020 (https://hackingthe.cloud/aws/exploitation/ec2-metadata-ssrf/).

[6] Check Brian Kerbs’ report on this breach: “Capital One Data Theft Impacts 106M People,” Krebs On Security, July 30, 2019 (https://krebsonsecurity.com/2019/07/capital-one-data-theft-impacts-106m-people/comment-page-3/#comment-493895) and “What We Can Learn from the Capital One Hack,” Krebs On Security, August 2, 2019 (https://krebsonsecurity.com/2019/08/what-we-can-learn-from-the-capital-one-hack/). Check also Evan Johnson’s technical analysis of the breach: “Preventing The Capital One Breach,” published on his personal blog (https://ejj.io/blog/capital-one).

[7] Eaton Zvere, “Hacking into Toyota’s global supplier management network,” EatonWorks Blog, February 6, 2023 (https://eaton-works.com/2023/02/06/toyota-gspims-hack/).

[8] Sanjay Dalal and Erik Wilde, “The Deprecation HTTP Header Field,” IETF Internet Draft, December 11, 2023 (https://datatracker.ietf.org/doc/draft-ietf-httpapi-deprecation-header/).

[9] Traceable, 2023 State of API Security, 2023 (https://www.traceable.ai/2023-state-of-api-security), p. 16.

[10] Alex Hern, “Company forced to change name that could be used to hack websites,” The Guardian, November 6, 2020 (https://www.theguardian.com/uk-news/2020/nov/06/companies-house-forces-business-name-change-to-prevent-security-risk).

6 API security by design

This chapter covers

Mitigating the risk of predictable resource identifiers

Designing secure pagination patterns

Constraining user input to prevent large payload attacks

Designing strict data models to mitigate the risk of data corruption attacks

Understanding the risks of exposing server-side properties in user input

Designing and enforcing secure user flows through the API

In February 2024, Trello, the popular project management platform, suffered a major data breach affecting 15 million users. The data leaked contained personal names, usernames, emails, and other account information. Surprisingly, the threat actor didn’t need to breach the system to obtain the data. How could this happen?

Trello had an endpoint that conveniently allowed users to find other users by email and connect with them. Upon finding an existing user, the API returned their full profile. In other words, the endpoint revealed excessive user personal information. Besides, the endpoint was unauthenticated, and it wasn’t properly rate-limited, which allowed the threat actor to query the API anonymously millions of times without being traced. In response to the breach, Trello required authentication to access user profiles. This allows Trello to trace each request to a specific user account, and hence detect and flag abnormal user behavior.

The Trello breach highlights the impact of API design choices in our security posture. By combining lack of authentication with excessive data exposure, Trello created a highly sensitive business flow that was easy to abuse. And Trello isn’t an isolated case. Most APIs contain vulnerable designs. Sometimes, vulnerable designs serve a business purpose and are intended. For example, if we want to allow users to filter items in a product catalogue using random keywords, we won’t be able to apply any constraints on the search filter. This opens our API to attacks like SQL injection, schema enumeration, large payloads, and more. In such cases, we handle the potential threats in the implementation.

However, most API design vulnerabilities aren’t intended or justifiable. For example, many APIs expose sensitive data for no good reason, allow users to fetch millions of items in a single request, modify the status of an order or a payment, or game the system with fake product reviews and such. Sadly, most of these vulnerabilities cannot be addressed at runtime. For example, the vulnerabilities in the Trello API could only be fixed by changing the design. That means that to deliver secure APIs, we must focus on the design. As we learned in chapter 3, we must shift our perspective on API security to the left and tackle potential issues from the very beginning of our API development process. That’s what this chapter is all about.

In this chapter, you learn to identify unintended design vulnerabilities in your API designs and address them properly. Over the past years, I’ve had the opportunity to review hundreds of APIs, and I’ve put together a list of common categories of design vulnerabilities, including predictable identifiers, unconstrained input, flexible schemas, exposing server-side properties in user input, sensitive data exposure, and unsafe user flows. In this chapter, we’ll go through each of these categories in detail, we’ll see practical examples and how threat actors exploit them, and how to fix them. Let the journey begin!

6.1 What is vulnerable API design?

APIs can be vulnerable or secure by design. But what does this mean in practice? To understand how API design affects our security posture, we must learn to look at the API from an attacker’s perspective. Threat actors look for ways to trigger undesired or unexpected behaviors in our system. Sometimes, they do this by breaking through our authentication and authorization systems, tampering access tokens, or taking over other user accounts. Most threat detection systems will easily flag some of those attacks, and many web application firewalls (WAFs) will successfully block them.

In practice, most threat actors will try to get around your WAF and your threat detection systems by looking like normal users. According to Salt Security’s State of API Security 2025, 98% of API attacks are authenticated (https://www.infosecurity-magazine.com/news/99-organizations-report-api/ and https://content.salt.security/state-api-report.html), and as Paulo Silva, maintainer of the OWASP Top 10 for APIs, highlights, most hackers will be careful to stay below your rate-limiting threshold (Paulo Silva and Erez Yalon, “OWASP API Security Project,” OWASP Global AppSec, 24-28 June 2024, Lisbon). This means that, for all intents and purposes, hackers look like normal users. They’re looking for design vulnerabilities in your system that they can exploit to their benefit.

What do design vulnerabilities look like and how do threat actors exploit them? Imagine we have an e-commerce API with a product catalog. As shown in figure 6.1, to browse the catalog, we expose a GET /products endpoint. Since we have millions of products, we won’t send the whole list in one go, since that would put overwhelming pressure on our database and our servers. Instead, we’ll allow users to request a small chunk of the list at a time. For example, 50 items per request. This is what we call pagination, and it’s a fundamental feature in endpoints that represent collections of resources.

DEFINITION

Vulnerable API design refers to vulnerabilities that an API has by design. We say by design because such vulnerabilities must be tackled at design time. It includes things like predictable identifiers, improper pagination, and unconstrained user input. Threat actors exploit design vulnerabilities to cause unexpected behaviors in our APIs by sending malicious requests that look legitimate by design. Most API design vulnerabilities can be mitigated in the implementation, however doing so creates a drift between the implementation and the design, which makes the API more difficult to test and understand. To effectively tackle API design vulnerabilities, we must go back to the beginning of the process and address them at design time.

Figure 6.1 We use pagination to slice a large collection of items into smaller chunks. For example, on an e-commerce website, we use pagination to retrieve a few products at a time when browsing the catalog.

There are different patterns for implementing pagination, such as page-based pagination, offset-based pagination, cursor-based pagination, and so on.[1] Listing 6.1 shows an OpenAPI specification for the GET /products endpoint using page-based pagination. We expose the following parameters to paginate the results:

page: allows the user to select the page they want to look at

perPage: allows the user to select how many items per page they want to see

Listing 6.1 Vulnerable pagination

paths:
  /products:
    get:
      parameters:
        - name: page
          in: query
          required: false
          schema:
            type: integer
        - name: perPage
          in: query
          required: false
          schema:
            type: integer

The example in listing 6.1 shows a standard and well-defined pagination pattern, but it is also a great example of vulnerable design. How is that? Let’s look into the query parameters. From the specification, we know page and perPage are both integers and not required. Presumably, they have reasonable default values, although that’s not documented in the specification. The expectation is that users will assign sensible values to the parameters, for example by requesting 10 items per page as in GET /products?perPage=10, or by requesting page 5 as in GET /products?page=5.

However, notice how listing 6.1 doesn’t specify minimum or maximum values for the parameters. As illustrated in figure 6.2, this means that, by design, a user could request 100 million items in a single request, as in GET /products?perPage=100000000. Our system may be able to handle one of those requests flawlessly, but what about 100 concurrent requests like that? At some point, our database will come under pressure and bring our website down, opening the door for denial of service (DoS) attacks. And how about users request page -1 (GET /products?page=-1), or 0 items per page (GET /products?perPage=0)? Depending on the implementation details, we may get anything from internal server errors to unexpected items in the list.

Figure 6.2 Threat actors can exploit unconstrained pagination parameters to trigger unexpected behaviors in our server or even shut them down. For example, if a threat actor requests 100 million items from our API, the request will cause performance problems in our database.

Now, the implementation could handle these problems at runtime. For example, it could default to a lower value when perPage’s value goes above 100. However, this is delegating crucial security decisions to the will of developers. And as much as developers are conscious of security, they’re also under immense pressure to deliver their code on time. All this means that security gets easily overlooked if we don’t tackle it by design.

To make the pagination pattern in listing 6.1 secure by design, we want to add minimum and maximum values to the query parameters, as shown in figure 6.2.

Listing 6.2 Secure pagination

paths:
  /products:
    get:
      parameters:
        - name: page
          in: query
          required: false
          schema:
            type: integer
            minimum: 1
            maximum: 100
        - name: perPage
          in: query
          required: false
          schema:
            type: integer
            minimum: 10
            maximum: 100

Notice how listing 6.2 includes a maximum value for the page parameter. This means users won’t be able to look for products beyond page 100. Does that make sense? It totally depends on the use case. If you want your users to be able to scroll indefinitely, then you shouldn’t include a maximum value. But if you want to prevent them from scraping your whole database of products, a maximum number of pages is a good way to limit their scrolling by design.

The best part about the design in listing 6.2? Now we can use contract-testing and fuzz testing tools to validate that our implementation abides with the implementation. We’ll learn more about these testing strategies in chapter 12. For now, watch for unbound pagination parameters, and ensure they have sensible constraints.

The pagination example is a simple illustration of how API design impacts our security posture. Now that we have an idea of how it works, let’s dive into more complex categories! We’ll begin by warming up with an easy one: predictable identifiers.

6.2 Predictable identifiers

Most APIs are resource-oriented, meaning they’re designed around resources. For example, as illustrated in figure 6.3, a payments API represents each payment as a resource, an orders API represents each order as a resource, and so on.

Figure 6.3 Most APIs are designed around resources. For example, a payments API represents each payment as a resource, while an orders API represents each order as a resource too.

Each resource has an identifier (ID), and we use those ID values to look for and retrieve the resources from the server. For example, a payment with an ID of 1 can be represented by a URL like /payments/1. The problem is when resource identifiers follow a predictable pattern. For example, as illustrated in figure 6.4, if a user makes a payment and gets back an ID of 1, and then makes a second payment and gets back an ID of 2, there’s clearly a pattern of incremental integer IDs here. The next natural step is to play with different identifiers in the resource URI path (/payments/{payment_id} in this case) and see if we can get our hands on resources that don’t belong to us and we shouldn’t be able to access. We call the practice of looking for new resources in a server by manipulating their identifiers in the resource URI resource enumeration.

Figure 6.4 When an API uses a predictable pattern to identify resources, such as incremental integer IDs, threat actors will play with those values to try and access resources that don’t belong to them.

Incremental integer IDs are a very common type of primary key or record identifier used in SQL databases. They’re natively supported, they’re efficient, and easy to work with. Using incremental IDs in our database isn’t bad. The problem is when they get exposed directly through the API, as in figure 6.4. If that happens, we give threat actors a recipe of how to scan for resources in our server. Paired with a weak implementation of authorization and access controls, sequential IDs can lead to broken access level authorization (BOLA) attacks and data breaches.

For example, in April 2021, ClubHouse, the popular social media platform, suffered a leak of 1.3 million user records, including personal names, usernames, Twitter (now x.com) handles, Instagram handles, and more (https://cybernews.com/security/clubhouse-data-leak-1-3-million-user-records-leaked-for-free-online/). ClubHouse used incremental integer IDs to identify each user, it also lacked rate-limiting on its users endpoint, and it didn’t protect the endpoint with user-based access controls. By exploiting these vulnerabilities, a threat actor was able to scrape millions of records from the API. ClubHouse responded to the news by clarifying it wasn’t hacked and the API was used as intended. In other words, the API was vulnerable by design.

To deter threat actors from discovering resources in your server, don’t expose predictable IDs in your API. You can do this at the database or the application level. If your database supports it, you can use string-based identifiers such as globally unique identifiers (GUIDs) or universally unique identifiers (UUIDs, especially UUIDv4) as your primary key. Identifiers like GUIDs and UUIDs don’t follow a sequential pattern, and that mitigates the risk of resource enumeration. If you must use incremental IDs in your database for legacy or other technical reasons, consider using surrogate public IDs. A surrogate public ID is a public-facing identifier, such as a UUID, that maps internally to your incremental IDs. The idea is to expose the public surrogate ID to your API consumers, while internally using the incremental ID for table joins and other references. Alternatively, you can also use a custom encoding method, such as Sqids (pronounced “squids”, https://sqids.org/), which allows you to generate random unique IDs from numbers. The idea is to take the incremental ID from your database and transform it into another value that threat actors can’t trace back to the original identifier, hence mitigating the risk of resource enumeration.

One word of warning: using non-predictable identifiers doesn’t mean your API suddenly becomes immune to BOLA and resource enumeration. It simply makes it more difficult to scan your server for resources. There are signals threat actors can still leverage to discover resources, such as differential response latencies, different status codes, or subtle differences in the messaging for existing and non-existing resources. You must also have robust authorization checks on every resource endpoint and a sensible configuration of rate-limiting policies.

Now that we understand the risks associated with predictable identifiers, let’s move on to something more complex: unconstrained user input.

6.3 Unconstrained user input

Most API operations require input from users. For example, when we place an order online, we specify the products we want to buy and how many; when we make a payment, we submit our credit card details; when we book a ride, we select our destination and the type of vehicle we want. These are all examples of write operations – they create resources in the server with the data we supply in the form of input, typically through request payloads.

Read operations also accept user input. For example, as illustrated in figure 6.5, when we check the status of our order, we provide the order ID. When we browse a product catalog, we provide pagination parameters as we saw in section 6.1. And when we want to obtain information about a certain location, we provide its latitude and longitude. Read operations usually take user input in the form of query and path parameters.

Figure 6.5 Most API operations require input parameters, including read operations. For example, GET /orders/1 has a URL path parameter, which is the order ID equal to 1.

The ability to capture user input in our APIs is what makes our applications come to life. It allows users to access the business logic behind the API. But it is also the main attack vector in the API. User input gets to interact with our business and data layers, and if it isn’t properly constrained or sanitized, it can wreak havoc in our system.

Unconstrained user input refers to the ability to insert random values in any parameter open to the user, including URL query and path parameters, request payloads, and request headers. As shown in figure 6.6, many APIs sit behind a browser or mobile application, and in those cases, we often delegate the job of constraining user input to the browser or mobile application. This gives a false sense of security, since nothing prevents threat actors from accessing the API directly and sending malicious content.

Figure 6.6 Many applications delegate the responsibility of constraining user input to the user interface (UI). In this example, the browser ensures that users can only select 10, 20, or 50 items per page when browsing a product catalog.

What exactly does unconstrained input look like? In section 6.1, we saw an illustration of unconstrained input with pagination parameters. In that case, the lack of bounds in the page and perPage parameters allows a threat actor to bring the system down and trigger unexpected behaviors. Let’s see other examples.

Imagine we have an application where users can submit book reviews, something along the lines of Goodreads. The application has an API with a POST /books/{book_id}/reviews endpoint, which allows us to submit reviews for a specific book with the schema shown in listing 6.3.

Listing 6.3 Unconstrained book review schema

POST /books/{book_id}/reviews

BookReview:
  type: object
  required:
    - rating
  properties:
    rating:
      type: integer
    review:
      type: string

The schema has two properties: rating and review. rating is required and it’s an integer, while review is optional and it’s a string. Both parameters are also unconstrained. In this case the browser application would constrain user input by allowing users to select a rating between 1 and 5.

However, by design the API isn’t enforcing any constraints, and a threat actor could send a higher rating directly to the API, for example the following request:

POST /books/1/reviews
{"rating": 500}

With this request, the threat actor can increase the average rating of a book. Similarly, a negative rating could be used to push down the average rating of other books. By engaging with the API directly and bypassing the constraints in the browser application, threat actors can abuse the system and manipulate the ratings of certain books.

The BookReview schema is also vulnerable to large payload attacks. How is this a problem? Imagine there’s a competitor book on the website, and we don’t want readers to be able to access the competitor’s book details. To accomplish that, we write reviews with very long texts. Sure enough, one of those reviews won’t cause too much trouble. However, as illustrated in figure 6.7, a few hundreds of those reviews will make it hard to load those records from the database and serve the request, making our API vulnerable to resource exhaustion attacks.

Figure 6.7 Unconstrained strings allow users to send very large payloads to our server. In this example, a threat actor creates a few reviews with very large text for a book with ID 1. When a user tries to retrieve the book later on, the server is unable to serve the request since it can’t load so many large records at once.

To fix this problem by design, we include constraints in the BookReview schema. As shown in listing 6.4, we constrain the values of rating to 1-5, and allow a maximum of 2000 characters for the review property. Now that our API is secure by design, threat actors can’t abuse our review system anymore!

Listing 6.4 Safe book review schema with constraints

POST /books/{book_id}/reviews

BookReview:
  type: object
  required:
    - rating
    properties:
      rating:
        type: integer
        minimum: 1
        maximum: 5
      review:
        type: string
        maxLength: 2000

What about query and path parameters? Are they also vulnerable? Just as much. We already saw an illustration of vulnerable query parameters in the pagination example in section 6.1. That example included unbound integers. Let’s now see an example of unconstrained strings. Imagine the GET /products endpoint from listing 6.2 now includes two additional parameters: filter and sortBy, as shown in listing 6.6. Both are strings without constraints, and both are non-required.

Listing 6.5 Insecure string query parameters

paths:
  /products:
    get:
      parameters:
        - name: page
          in: query
          required: false
          schema:
            type: integer
        - name: perPage
          in: query
          required: false
          schema:
            type: integer
        - name: filter
          in: query
          required: false
          schema:
            type: string
        - name: sortBy
          in: query
          required: false
          schema:
            type: string

When using the GET /products endpoint, the expectation is users will choose sensible values for the filter and the sortBy parameters. For example, filtering by toasters, as in GET /products?filter=toasters, and sorting by price as in GET /products?sortBy=price. However, threat actors will use malicious values too, like SQL injections, as in the following request: GET /products?filter=' OR 1=1--. As shown in figure 6.8, if our database queries aren’t parametrized, the SQL injection will go in and disable any filters and constraints in our transaction. In some situations, this could lead to broken object level authorization and other vulnerabilities.

Figure 6.8 Unconstrained string parameters allow threat actors to send SQL injection attacks to our servers. In this example, a threat actor disables any filters and constraints in our query by setting the filter query parameter to ' OR 1=1; --.

Another SQL injection example that would work here is this:

GET /products?filter=' AND 3133=(SELECT 3133 FROM PG_SLEEP(10))--

This SQL injection appears more complicated, but it’s really just telling the database to wait for 10 seconds before returning the data. One request like this won’t pose much problem, but a few thousand concurrent requests like this will eventually use up all the available resources in our connection pool and make the API unable to serve any further requests involving the database.

The unconstrained sortBy parameter is also vulnerable to schema enumeration attacks. As shown in figure 6.9, we typically sort items by one of the columns in our database schema. Without constraints on sortBy, threat actors can throw different values to discover hidden columns of our product table. Schema enumeration attacks can lead to disclosure of sensitive data and configuration in our system.

Figure 6.9 The sortBy parameter allows us to sort items by properties of the Product model, like name, price, and rating. Since sortBy is unconstrained, a threat actor tries other values. Sorting by non-existing fields like asdf throws a server error, but sorting by is_exclusive return successfully, hence allowing the threat actor to discover hidden properties of the model.

How do we mitigate the risks of unconstrained string parameters? As illustrated in listing 6.6, in the case of sortBy, a good strategy is using enumerations. We want to allow users to sort items using a restricted number of values, such as price, review, and name, and an enumeration is the right solution for that. This will prevent threat actors from using random values by design.

Listing 6.6 Constraining sortBy with an enumeration

paths:
  /products:
    get:
      parameters:
        [...]
        - name: sortBy
          in: query
          required: false
          schema:
            type: string
            enum:
              - price
              - review
              - name

What about the filter parameter? In this case, we probably want to allow users to filter by random words, which means we can’t easily apply restrictions on the schema. This is a situation in which the business use case justifies a vulnerable design. What we want to do in this situation, is handle the problem at runtime, by parametrizing all our queries and hence removing the risk of SQL injection. Another helpful tool in this case is a web application firewall (WAF), which in most cases will be able to fend off request containing malicious injection attacks.

Now that we understand how unconstrained input makes our APIs vulnerable, let’s take a look at a more complex problem: flexible schemas!

6.4 Flexible schemas

Flexible schemas are schemas that allow us to compose payloads with different combinations of properties. This can happen when properties are optional, as in the review property of the BookReview schema in listing 6.3, and when we allow the presence of undeclared properties. Let’s look at each of these cases in detail.

6.4.1 Optional properties

Optional properties are non-required properties in a schema. We include optional properties to give users flexibility. For example, in the BookReview schema, we give users the choice of including a written review of the book, but this isn’t required.

Another use for optional properties is when we want to represent two or more data models with the same schema. It usually happens because we have data models with very similar properties and one or two properties specific to each model.

For example, imagine we have an online bookstore that sells eBooks and printed books. Both book formats represent different types of entities. However, since they both share many properties in common, like author and title, we may decide to represent them with the same schema. As illustrated in figure 6.10, the risk here is data corruption. If someone mistakenly creates a printed book record with eBook properties, this will create problems in other parts of the system when we process the data. For example, in figure 6.10, the book ends up written in the database with its pages field set to null. When retrieving a list of books, we may have a feature that dynamically estimates the number hours it takes to read the book given the reader’s average reading speed and the book’s size, using pages in the case of printed books and byte_size in the case of eBooks. Since the newly created printed book record does not contain a valid value for pages, the endpoint fails when it tries to calculate its reading time. In the hands of a threat actor, this is a perfect recipe to wreak havoc on our application and our business.

Figure 6.10 A threat actor sends an invalid representation of a printed book’s, hence corrupting the representation of that book in the database and making it impossible for other users to retrieve that record.

To understand data corruption vulnerabilities, let’s expand on the printed vs eBooks example. The only difference between eBooks and printed books is that one measures the length of the book in bytes, while the other does it in pages. Since all the other properties in the model are the same, including author name and book title, we decide to represent both formats with the same schema, as shown in listing 6.7.

Listing 6.7 Representing multiple data models with the same schema

Book:
  type: object
  required:
    - author
    - title
    - format
  properties:
    title:
      type: string
    author:
      type: string
    format:
      type: string
      enum:
        - print
        - ebook
    pages:
      type: integer
    byte_size:
      type: integer

To distinguish between the two types of books, listing 6.7 introduces a property called format, which is an enumeration of print and ebook. When authors list their books on the website, we expect them to use pages if it’s a printed book, and byte_size if it’s an eBook. For example, to list a printed book, we expect a request body like this:

POST /books
'{"title": "Microservice APIs", "author": "Jose Haro Peralta", "format": "print", "pages": 440}'

And if listing an eBook, we expect a request body like this:

POST /books
'{"title": "Microservice APIs", "author": "Jose Haro Peralta", "format": "ebook", "byte_size": 15110000}'

However, as usual, nothing prevents threat actors from going directly to the API and sending bad payloads. For example, the following request creates a printed book record with byte_size instead of pages:

'{"title": "Microservice APIs", "author": "Jose Haro Peralta", "format": "printed", "byte_size": 15110000}'

To prevent data corruption attacks like this, we represent different data models with different schemas. If we want to save duplications by grouping all the common properties in a shared model, we can use composition, as shown in listing 6.8.

Listing 6.8 Using composition to handled shared model properties

BaseBook:
  type: object
  required:
    - author
    - title
  properties:
    title:
      type: string
    author:
      type: string

Ebook:
  allOf:
    - $ref: #/components/schemas/BaseBook
    - type: object
      properties:
        byte_size:
          type: integer

PrintedBook:
  allOf:
    - $ref: #/components/schemas/BaseBook
    - type: object
      properties:
        pages:
          type: integer

By representing each data model with its own schema, we can apply better validation rules on request payloads, hence mitigating the risk of data corruption attacks. Now that we understand how optional properties can pose a risk, let’s look at a more subtle and often misunderstood feature in JSON Schema: the allowance of additional properties.

6.4.2 Additional properties

Additional properties are properties that we haven’t declared as part of our schemas. For example, listing 6.9 represents a schema to place orders in an e-commerce API. The schema defines two required properties: product and quantity. Every request sent to the server must include both properties, as in the following example:

 POST /orders
'{"product": "3598abeb-2f93-4f3f-a14b-103f27c2c040", "quantity": 1}'

Listing 6.9 Flexible schema allowing additional properties

PlaceOrder:
  type: object
  required:
    - product
    - quantity
  properties:
    product:
      type: string
      format: UUID
    quantity:
     type: integer

As it stands, the PlaceOrder schema in listing 6.9 allows additional properties in the payload. For instance, the following request is perfectly acceptable:

POST /orders
'{"product": "3598abeb-2f93-4f3f-a14b-103f27c2c040", "quantity": 1, "foo": "bar"}'

Allowing the presence of additional properties opens the door to many kinds of abuses in our system. For example, let’s say the request payload in listing 6.9 has a corresponding response payload with the shape in listing 6.10. Listing 6.10 introduces two new properties: id and status.

Listing 6.10 Response payload for POST /orders

Order:
  type: object
  required:
    - status
    - id
    - product
    - quantity
  properties:
    id:
      type: string
      format: UUID
    status:
      type: string
      enum:
        - placed
        - paid
        - delivered
        - returned
    product:
      type: string
      format: UUID
    quantity:
      type: integer

Since PlaceOrder in listing 6.9 doesn’t ban additional properties, a threat actor can leverage the information from listing 6.10 to send the following request:

POST /orders
'{"product": "3598abeb-2f93-4f3f-a14b-103f27c2c040", "quantity": 1, "status": "paid"}'

As you can see, the flexible model in listing 6.9 allows threat actors to manipulate the state of their orders. This can allow them to bypass payments, obtain refunds for orders they haven’t paid for, and much more.

Why is PlaceOrder in listing 6.9 vulnerable to this type of attack and what can we do about it? To understand this vulnerability, we must understand the nature of model definitions in OpenAPI, the standard we use to describe REST APIs.

Note

The discussion in this section goes into very technical details of JSON Schema and OpenAPI. If you’re not very familiar with how JSON Schema and OpenAPI work, see “Documenting REST APIs with OpenAPI,” chapter 5 of Microservice APIs by Jose Haro Peralta (Manning, 2023) for a quick introduction. For an in-depth explanation of how OpenAPI and JSON Schema work, check out Joshua S. Ponelat and Lukas L. Rosenstock’s Designing APIs with Swagger and OpenAPI (Manning 2022).

OpenAPI is based on top of JSON Schema, which is a standard for describing data models. When we describe a data model in OpenAPI, like we do in listing 6.9, we’re using JSON Schema syntax. As you can see in listing 6.9, JSON Schema allows us to specify the type of the data model (object type in this case), enumerate its properties, and indicate which properties are required. The catch with JSON Schema is that, by default, models aren’t restrictive, that is they allow additional properties. That’s why PlaceOrder’s definition in listing 6.9 allows us to include additional fields in the payload, like status.

To make our schemas strict, we must use one of the following keywords: additionalProperties or unevaluatedProperties, as illustrated in figure 6.11. These two keywords control whether and how our model accepts additional or unknown fields. To ban additional fields, we set additionalProperties or unevaluatedProperties to false. Alternatively, we can use additionalProperties and unevaluatedProperties to allow unknown fields following a certain pattern or constraint. For example, to allow additional fields of string type, we use: "additionalProperties": {"type": "string"}.

The difference between additionalProperties and unevaluatedProperties is their application scope. additionalProperties operates within the model’s local scope, while unevaluatedProperties applies across multiple models. This difference is relevant if our models use composition (see an example in listing 6.8). Composition allows us to reuse models and combine their definitions via polymorphism, which is JSON Schema’s way of implementing inheritance (https://json-schema.org/blog/posts/modelling-inheritance). If we use composition, we can’t use additionalProperties, since it applies within the local scope of the model, and hence setting it to false would cause contradictions between all the models. To ban unknown fields using model composition, we use unevaluatedProperties. For simple models that don’t use composition, setting additionalProperties to false works just fine.

Figure 6.11 We use additionalProperties in a model to ban the presence of undeclared properties. In this example, the only allowed properties are author and title, and any additional properties, like rating, make the payload invalid.

As shown in figure 6.11, when we declare additionalProperties within a model, it bans the presence of any additional properties to that model, hence invalidating other models within a composed schema.

To prevent threat actors from manipulating the state of their orders, we set additionalProperties to false in the PlaceOrder, as shown in listing 6.11. With this simple change, our data models will only allow the presence of declared properties, like product and quantity in this case. Additional properties in the request payload result in a 422 response.

Listing 6.11 Banning additional properties with additionalProperties

PlaceOrder:
  type: object
  required:
    - product
    - quantity
  properties:
    product:
      type: string
      format: UUID
    quantity:
      type: integer
  additionalProperties: false

Now that we know about the risks of flexible data models and how to handle them, let’s move on to a more subtle vulnerability: exposing server-side properties in user input!

6.5 Exposing server-side properties in user input

All APIs have a concept of server-side or read-only properties. For example, as shown in figure 6.12, when we make a payment on a website, we get back information that allows us to track the payment, such as its identifier and status. The payment ID and its status are examples of server-side properties. The server manages them, and users must not be allowed to modify such properties. If we expose those properties in user input, threat actors may be able to manipulate them. Due to the nature of API designs, this is a fairly common vulnerability when we reuse the same data model for input and output. Let’s see how it works.

Figure 6.12 To make a payment, we indicate the amount and currency. The response from the API includes additional information, like the ID and status of the payment.

Every operation in an API has a request and a response, and usually, requests and responses carry data in the form of a payload. For example, as shown in figure 6.12, to make a payment, we send out our payment details to the server and get back a representation of the payment. The request and response payloads are different, but for convenience, we may decide to use the data model from listing 6.12 to represent both of them.

Listing 6.12 Data model for a payment

Payment:
  type: object
  required:
    - currency
    - amount
  properties:
    id:
      type: string
      format: uuid
    amount:
      type: number
    currency:
      type: string
    status:
      type: string
      enum:
        - pending
        - accepted
        - rejected
        - sent

The Payment model in listing 6.12 has two required properties: currency and amount. This is the information we send to the server to make a payment. The optional properties are id and status, which are server-side properties. By making server-side properties optional, we can use the same model to represent input and output, as shown in listing 6.13.

Listing 6.13 Reusing a schema to represent input and output in a payments API

paths:
  /payments:
    post:
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Payment'
      responses:
        '201':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Payment

Listing 6.13 uses the same model to represent the POST /payments request payload and its response. By reusing the same model for input and output, we save ourselves the effort of maintaining two different models. This is convenient, but it comes at a risk: we’re exposing server-side properties in user input. The expectation here is that users will use the API as intended. When users make a payment, they’re expected to send requests with the following shape:

POST /payments
'{"amount": 100, "currency": "USD"}'

As illustrated in figure 6.13, in many cases, our APIs sit behind browser-based or mobile user interfaces (UIs), and we often delegate to those applications control over how users engage with the API. In this case, the UI ensures users send the right type of payload to the server. However, nothing prevents threat actors from going directly to the API and sending payloads like below:

POST /payments
'{"amount": 100, "currency": "USD", "status": "settled"}'

Since we’re exposing the same Payment model for request and response, threat actors get access to sensitive server-side properties like payment status. In situations like this, it’s all down to the implementation details of the server and how we handle payloads like this. As illustrated in figure 6.14, we may reject such payloads with a 422 status code, or we may use a data transfer object pattern that captures only the expected information from the request payload.

A better approach is to make the API secure by design by having different models for input and output. In this case, we want to create a different model to represent request payloads, as illustrated in listing 6.14. This time around, we create a MakePayment schema to represent the data required to make a payment through the POST /payments endpoint. MakePayment doesn’t expose server-side properties, hence making the API secure by design. We also make the model strict by setting additionalProperties to false. As we learned in section 6.4.2, this prevents threat actors from setting additional properties in the payload, making the model even safer.

Listing 6.14 Data model for a payment

paths:
/payments:
  post:
    requestBody:
      required: true
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/MakePayment'
    responses:
      '201':
        content:
          application/json:
            schema:
                $ref: '#/components/schemas/Payment
components:
  schemas:
    Payment:
      ...
    MakePayment:
      type: object
      required:
        - currency
        - amount
      properties:
        amount:
          type: number
        currency:
          type: string
      additionalProperties: false

With everything you’ve learned so far in this chapter, you’re in an excellent position to start creating secure APIs that mitigate the risk of exploits as much as possible. So far, we’ve focused on constraining user input at the level of individual operations. But API operations come together into user flows, and it’s often the design of user flows that open a flaw in our security posture. In the next section, we discuss how we design secure user flows through our APIs.

6.6 Design safe user flows

API operations don’t stand alone, we use them in the context of a user journey or flow. We use APIs to buy items online, book rides, buy concert tickets, and so on. These are all examples of user or business flows. According to Imperva’s 2024 State of API Security Report, the lion’s share (27% of all) of API attacks target sensitive business flows (https://www.infosecurity-magazine.com/news/business-logic-abuse-api-attacks/). Sensitive business flows are a growing area of concern in API security. We are only starting to understand how to protect and monitor such flows, and there’s much work to be done in this field.

In section 4.6, we learned to protect sensitive business flows by using CAPTCHA challenges, rate-limiting requests, and other strategies. In this section, we turn our attention to design. The questions we try to answer are: how do we design business flows that are less likely to be exploited? How do we document, enforce, and monitor such flows? We’ll answer these questions by proposing a model that leverages elements of API design, authentication, and observability. This model is technology-agnostic, and I’ve left implementation details out of the discussion to make the model applicable in a wide variety of scenarios. When implementing the measures discussed in this section, evaluate carefully how well they fit your particular needs and the constraints of your infrastructure and technology stack.

Let’s begin by analyzing how user flow design impacts our API security posture by using the example of an e-commerce application.

Figure 6.13 When buying products from an e-commerce website, the user interface guides the customer through the desired journey, from browsing the product catalog to the check out.

As illustrated in figure 6.13, buying a product on an e-commerce site like Amazon typically involves a few steps like

browsing the online catalog for the product we’re looking for

checking offers from different vendors, comparing product specifications and customer reviews

adding the products we want to buy to our basket

proceeding to checkout, applying discount codes if we have any, and making a payment

Each of the steps in this user flow connects to one or multiple API operations. For example, landing on the homepage catalog triggers a request to the GET /catalog endpoint, and adding items to the cart triggers a request to the POST /carts endpoint. A customer using our website will follow all the steps in figure 6.13, and the user interface (UI) will guide them through that journey. Since the UI controls how a user interacts with our website, it helps us enforce a certain flow.

However, a user going directly to the API can skip steps in this flow. For example, as shown in figure 6.14, if a user already knows the ID of a product, they can go straight to step 3 by sending a request to the POST /carts endpoint. From a security point of view, the question is whether we should allow this.

Figure 6.14 By going directly to the API, a threat actor can jump steps in the user journey and go straight to adding items to their cart.

Figure 6.15 shows a threat model for the user flow in figure 6.14. It’s clear that, by allowing users to skip steps in the flow when going directly to the API, we’re opening our application to abuse. In this case, threat actors can exploit the flow in figure 6.14 to buy out the whole stock of a product and resell it at a higher price using automated bots. As we learned in section 4.6, this is known as scalping, and it results in unhappy customers.

Figure 6.15 By skipping steps in the user journey, threat actors can leverage automation to launch scalping attacks against our e-commerce site and buy out the whole stock of a product ahead of real customers.

Nobody wants unhappy customers, so how do we prevent this type of abuse against the API? How do we tell a scalper bot from a legit customer when a request arrives in our API? The first step is to identify our objective. In this case, what we really want is give human users a fighting chance against the bots. We’ll not get rid of the bots completely, let’s be clear about that. But we can force the bots to behave like humans, and at that point, it may not be worthwhile for threat actors to run bots against our site anymore.

A key difference between the legit customer and the bot is the different flows they’ll follow. Customers will follow the flow in figure 6.13 since that flow is enforced by our UI. Bots, on the other hand, will take shortcuts and use the flow in figure 6.14. Also, human customers will take a few seconds to move from one step to the next, while bots will complete the whole flow in microseconds. The actionable points are:

Enforce a specific flow in the API

Track the latency between steps

How do we enforce user flows in the API? The first step is to formally describe the desired user flow. To help organizations describe user flows through their APIs, the OpenAPI Initiative recently launched the Arazzo Specification (https://github.com/OAI/Arazzo-Specification). Many APIs contain hundreds of endpoints, and it’s not clear what sequence of steps users need to follow to accomplish certain goals. Arazzo fills this gap.

Note

The Arazzo specification is a new standard developed by the OpenAPI initiative to help organizations describe the user flows that their APIs support. You can find examples for how to document user flows on Arazzo’s official GitHub repository: https://github.com/OAI/Arazzo-Specification/tree/main/examples. To learn more about Arrazzo, check out the official documentation (https://spec.openapis.org/arazzo/latest.html), and Frank Kilcommin’s presentation “The Arazzo Specification”, presented at APIDays London, 18-19 September 2024 (https://youtu.be/Lrsl5KV9Fuw). Kilcommins is Arazzo’s lead developer and API evangelist at SmartBear.

By helping us describe user flows, Arazzo can also help us enforce them. Let’s say we want to ensure users go through the following steps when interacting with the API:

User lands on product detail page -> GET /products/{product_id}

User adds products to the cart -> POST /carts

User calculates price -> POST /carts/{id}/price

User selects a delivery slot -> POST /carts/{id}/delivery-slot

User pays -> POST /carts/{id}/payment

Listing 6.15 shows how we document this flow using Arazzo. A flow can consist of steps from multiple API specifications, and we list those specifications under the sourceDescriptions field. Under the workflows field, we list the steps users must go through in order. Each step has an ID (workflowId) and an operationId. The operationId corresponds to the endpoint’s operation ID in the OpenAPI specification. In each step, we specify the required input parameters and the successful responses, and if applicable, how the response from one operation feeds information into the next. For instance, the create-cart step returns a cart_id that we can replace in the input parameter for the calculate-price operation.

Listing 6.15 Data model for a payment

arazzo: 1.0.0
info:
  title: Checkout flow
  version: 1.0.0
sourceDescriptions:
  - name: orders
    url: ./orders.yaml
    type: openapi
workflows:
  - workflowId: checkout
    steps:
      - stepId: product-detail
        operationId: product-detail
        parameters:
          - name: product_id
            in: path
        successCriteria:
          - condition: $statusCode == 200
      - stepId: create-cart
        operationId: create-cart
        parameters:
          - name: product_id
            in: body
        successCriteria:
          - condition: $statusCode == 201
        outputs:
          cart_id: $response.body.id
      - stepId: calculate-price
        operationId: calculate-price
        parameters:
          - name: cart_id
            in: path
            value: $steps.create-cart.outputs.cart_id
        successCriteria:
          - condition: $statusCode == 200
      - stepId: delivery-slot
        operationId: book-delivery-slot
        parameters:
          - name: cart_id
            in: path
            value: $steps.create-cart.outputs.cart_id
        successCriteria:
          - condition: $statusCode == 200
      - stepId: payment
        operationId: make-payment
        parameters:
          - name: cart_id
            in: path
            value: $steps.create-cart.outputs.cart_id
        successCriteria:
          - condition: $statusCode == 201
        outputs:
          payment_id: $response.body.id
    outputs:
      payment_id: $steps.payment.outputs.payment_id

Now that we have a formally described flow, we can enforce it in the API. Since this is a highly sensitive business flow, we’ll require authentication in all the steps. This rules out attacks from unauthenticated bots and allows us to easily track user behavior more easily. Use a tracker, such as a session-based flow tracking cookie, to monitor user interactions throughout the flow, and ensure users can only have one active tracker session at a time. Combine this with detailed logs about user activity throughout the flow to be able to detect malicious behavior, as explained in chapter 11.

When we talk about enforcing this flow, we don’t mean we expect users to go through all those steps without interruptions. Legitimate users constantly deviate from the expected flow. For example, as illustrated in figure 6.16, a user may add a product to the cart, then check another page from our website unrelated to the checkout flow, and finally proceed to payment. This is a perfectly acceptable flow. In fact, a bit of randomness in the user flow, such as this one, is a good indicator that the requests are not coming from a bot. Figure 6.16 illustrates how we capture the steps taken by the user in their journey through our API in a flow tracker. Every time the user hits an API endpoint, we record the step in the flow tracker.

Figure 6.16 Real user interactions with our API follow a specific flow, but they may also include some deviation steps. In this diagram, a customer takes a few steps outside the expected flow, like checking the terms and conditions and additional product detail pages. We capture the user journey in a flow tracker.

The way we want to enforce flows is: when a user calculates the price, did they add a product to the cart previously? And did they check the product page before that? In other words, from step 2 onwards, to go through a step in the flow, the user must have gone through the previous step. We also want to track the time it takes the user to move from one step to the next. Human users are likely to move in seconds, while bots will move in milliseconds. Do your own research to determine reasonable values for your own website, and bear in mind that these values are likely to change over time. Finally, when the user completes the flow, we reset the tracker.

Eventually, bots will catch up with the expected user behavior, following the right flow with the expected time latencies between steps. This is not a bad thing, and it doesn’t mean that you’ve lost the war to the bots. It means you’ve obliterated the bots and forced them to behave like humans. It means your legitimate, human users stand a fighting chance of using your application the way it’s meant to. You can add additional anti-bot protection measures, like CAPTCHAs in critical steps of the flow. And since the flow is authenticated, you know who’s done what, and you can use this information to build a risk profile of every user. In highly sensitive flows, you can combine the measures described in this section with non-repudiation methods, such as mTLS (see section 7.5) and message signing (see section 10.6).

The illustration in figure 6.16 is implementation agnostic and it can be done in multiple ways. In chapter 11, we’ll go through an implementation example. For now, review the user flow designs of your APIs and document the expected interactions using the Arazzo specification. Make sure you apply all the constraints and mitigations we’ve discussed in this chapter. We’ll use all this information in chapter 12 to ensure that the implementation is secure and good to go.

6.7 Summary

Vulnerable API design refers to vulnerabilities that an API has by design. It includes things like predictable identifiers, improper pagination, and unconstrained user input. Threat actors exploit design vulnerabilities to send malicious requests that look legitimate. Since they look legitimate, such attacks are difficult to identify and trace.

Predictable resource identifiers, like auto-incremental integer IDs, give threat actors a pattern to scan our server for resources. To prevent it, mask the value of those identifiers, or use a different type of identifier in your data source.

Unbound pagination parameters allow threat actors to cause unexpected behavior in our system and request large amounts of items, putting pressure on our system. To prevent it, add abounds to pagination parameters.

Unconstrained string parameters allow threat actors to perform schema enumeration and SQL injection attacks. To mitigate this risk, constrain strings with enumerations and parametrize database queries.

Representing multiple models under the same schema with optional properties opens the door to data corruption attacks. To prevent them, represent different data models with different schemas.

Allowing additional properties in request payloads allows threat actors to perform mass assignment attacks. To prevent them, make schemas strict by setting additionalProperties or unevaluatedProperties to false.

Using the same schema to represent input and output allows threat actors to override sensitive properties in our system. To prevent it, define different schemas for input and output.

APIs expose business flows that allow users to buy items online, book rides, etc. We say such flows are vulnerable when threat actors can exploit them to produce undesired outcomes for our business. To mitigate the risk of business flow abuse, we enforce the flow through the API and track API interactions.

The Arazzo specification is a robust standard for describing API flows. It allows us to describe the steps a user must follow to complete a flow, and the relationships between the steps. By documenting our flows with Arazzo, we make it easier to enforce such flows through the API.

[1] For a full discussion, see “Pattern: Pagination” by Olaf Zimmermann, Mirko Stocker, Daniel Lübke, Uwe Zdun, and Cesare Pautasso, published on Patterns for API Design (https://www.microservice-api-patterns.org/patterns/quality/dataTransferParsimony/Pagination.html).

7 API authorization and authentication

This chapter covers

The role of authentication and authorization in API security

Best practices for working with JSON Web Tokens

Understanding Open Authorization (OAuth) and when to use each OAuth flow

Hardening security with sender-constrained tokens

Securing user identities with OpenID Connect

Leveraging role-based access controls to define sets of permissions

In August 2024, cybersecurity firm Bitdefender revealed that Solarman, one of the world’s largest photovoltaic monitoring and management platforms, was vulnerable to account takeover. By gaining access to other user accounts, threat actors could steal personal data and disrupt the supply of electricity (https://www.bitdefender.co.uk/blog/labs/60-hurts-per-second-how-we-got-access-to-enough-solar-power-to-run-the-united-states/, see also the more detailed technical report: https://blogapp.bitdefender.com/labs/content/files/2024/08/Bitdefender-PReport-solarman-creat7907.pdf). Solarman exposes APIs that allow manufacturers of photovoltaic monitoring devices to log their data. According to Bitdefender’s findings, Solarman’s API failed to validate access tokens correctly, allowing threat actors to forge tokens.

The Solarman story brings up a recurrent theme in the API security space: authentication and authorization are hard. Authentication is the process of verifying user identity, while authorization is the process of validating access to a resource or operation. As you’ll learn in this chapter, there are different types of access controls, such as user-based and role-based. But before we can authorize user access, we need to know who the users are, which is where authentication comes into place.

In this chapter, we break down API authentication and authorization so you can prevent the Solarman mistake. You’ll learn how APIs authenticate users and authorize their access, what are the most commonly adopted standards and the recommended best practices. In the next chapter, we’ll see practical coding examples everything we learn in this chapter into practice.

Authentication and authorization are the fundamental tenets of API security. If we fail to put together a robust process for managing and verifying user identities, and for authorizing their access to our APIs, threat actors will be able take over user accounts and steal their data. In previous chapters, we spoke about the importance of constraining user input, validating and sanitizing data, parametrizing database queries, and more. None of those security measures will protect us if we fail to authenticate and authorize users in the first place. Let’s see what it takes to do it!

7.1 Authentication vs authorization

Authentication and authorization are the first line of defense in an API. They are also the pillars of our security model, since all our identity-based access controls build on top of our authentication and authorization processes. As we saw in the Solarman example, a simple mistake in our authorization layer can overturn our entire security model, hence it is crucial that we understand well how authentication and authorization work and that we implement them correctly. In this section, we begin this effort by learning the difference between authentication and authorization and analyzing the role they play in API security.

Let’s begin by defining what authentication and authorization are: authentication is the process of verifying an identity, while authorization is the process of validating that such identity has access to a certain resource or operation. The use of the word identity in this definition is intended. API access can be granted to users and non-human identities alike. Non-human identities represent things like applications, services, and devices. We use non-human identities to enable machine-to-machine communication, third-party API integrations, and automation tasks that rely on APIs.

DEFINITION

An access token is a special element in an API request that proves that an identity has access to the API. Access tokens come in various forms and shapes, from opaque tokens to structured tokens such as JSON Web Tokens (see section 7.2).

Most APIs implement authentication and authorization with access tokens, but that’s not the only way. In the real world, many APIs also use API keys, cookies, mutual TLS authentication, and HMAC authentication. Let’s make a quick overview of them:

Access tokens are generally short-lived tokens that can be opaque or structured.

API keys are typically used with non-human identities and are generally valid for a longer period of time, like weeks or months. In any event, API keys must expire at some point, and your users must be allowed to rotate them.

Cookies are typically used with web applications and can include either opaque or structured tokens.

Mutual TLS (mTLS) authentication uses both the server and the client’s certificate to verify each other’s identities during the TLS handshake. In this case, each party encrypts traffic with their own certificate to prove its authenticity. This method is more commonly used in enterprise contexts to authenticate non-human identities like services, accounts, and applications (see section 7.5.1 for more on mTLS).

HMAC authentication uses a shared secret key between the client and the server, together with a hash function like SHA-256, to produce a digest based on elements of the API request, such as the URL, payload, and headers. The digest is then used to authenticate the request.

A less common but powerful authentication method is the secure remote password (SRP) protocol, which allows users and clients to authenticate without having to send their password to the authorization server. Instead, the user sends proof of their knowledge of a password using a hash and a salt. The SRP protocol was created at Stanford University and you can learn more about it on its website (http://srp.stanford.edu/project.html). If you’re interested in using this authentication method, you can use providers like AWS Cognito which implements SRP (check the official documentation: https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-authentication-flow-methods.html).

As illustrated in figure 7.1, authentication involves the exchange of credentials, typically in return for an access token.

Figure 7.1 During authentication, users exchange their credentials for an access token. During authorization, the API checks that the access token is valid, and that the user has access to the requested resources.

Once the user or non-human identity gets hold of the access token, they send it to our API to prove they have access to it. The API checks that the token is valid and that the identity has access to the requested resource or operation. That’s authorization. As illustrated in figure 7.1, there are two parts to the authorization process:

Verifying if the token is valid

Checking if the user is allowed to access the requested resource

To validate the token in the first step, we check if it comes in the right format, whether it has expired, has a valid signature, and so on. We’ll talk more about token validation in section 7.2.1. The second step involves validating that the user has access to the API, belongs to the right user group or role, has the right access scopes, and is authorized to perform the requested operation on a given resource.

For example, imagine a healthcare application in which patients can request appointments and communicate with their doctors, receptionists can manage the appointments, and doctors can access patient health records. Patients, receptionists, and doctors represent different user groups or roles with different levels of permissions and access scopes.

To access a patient’s records, a doctor must log in and obtain an access token. When the request arrives in the API server, the API must check whether the access token is valid, whether the user belongs to the doctors’ user group or role, and whether they are allowed to access the records of that particular patient. Similarly, the API must check whether a user belongs to the patient group or role before it allows them to request appointments and communicate with their doctors. The API must also check whether a user belongs to the receptionist group or role before it allows them to manage patient appointments.

This type of access validation must be done on every request, and it represents our first line of defense against threat actors. In the rest of the chapter, we break down the process of authenticating users and authorizing their access in more detail. We begin by studying the most common type of access token: JSON Web Tokens. Let’s go!

7.2 Understanding JSON Web Tokens

The most common type of token to authorize API requests is the JSON Web Token standard. JSON Web Token (JWT, pronounced ‘jot’) is a URL-safe representation format for JSON objects. It was proposed in IETF draft in 2012 authored by M. Jones, J. Bradley, and N. Sakimura (https://datatracker.ietf.org/doc/html/draft-ietf-oauth-json-web-token-00) and consolidated in 2015 into RFC 7519 (https://datatracker.ietf.org/doc/html/rfc7519).

JWTs are JSON objects that contain information about a user. For example, they tell us what APIs the user has access to, what user group or role they belong to, what access scopes they have, and what their identifier is. We call the information contained in those JSON objects claims.

Functionally, we distinguish between two main types of JWT: ID tokens and access tokens. We use ID tokens in the context of OpenID Connect (OIDC, more on OIDC in section 7.6) when the token’s subject is a human identity. ID tokens contain personal details (aka claims) about the user, such as their personal name, email address, date of birth, and so on. Access tokens contain claims about the right of a user or a non-human identity to access an API. This distinction is important: ID tokens carry user information; they don’t prove their right to access an API, and hence you must never use them to validate access to an API. To validate access to an API, we use access tokens. Now let’s see what JWTs look like.

7.2.1 Structure and representation of JSON Web Tokens

JSON Web Tokens contain three elements: a header, a payload, and a signature. As illustrated in figure 7.2, a JWT is the result of concatenating the base64url encoding of each element, using dots as the separator.

DEFINITION

Base64url encoding is a type of base64 encoding that is safe to use in URLs. Normal base64 encoding uses characters such as +, /, and =, which have specific meanings in URLs and filesystems and therefore aren’t URL-safe. Such characters would typically be percent-encoded into a URL. For example, the + character becomes %2B. However, this conversion would make base64 encoded strings unnecessarily long in URLs. Base64url encoding solve this problem by replacing the + and / characters by – and _ respectively, and omitting the = character altogether.

Figure 7.2 To produce a JWT, we base64url encode the header, payload, and signature, and we join them with dots as separator.

The JWT header is known as JSON Object Signing and Encryption (JOSE) header, and it is described in RFC 7515 (“JSON Web Signature (JWS)”, by M. Jones, J. Bradley, and N. Sakimura, May 2015, https://www.rfc-editor.org/rfc/rfc7515.html). The JOSE header contains metadata about the token, such as the algorithm and the key used to sign the token. The most commonly used parameters in the JOSE header are typ, alg, and kid, as illustrated in listing 7.1. Let’s see what they mean:

typ: the media type. In this case, it tells us that this is a JSON Web Token (JWT).

alg: the type of algorithm that was used to sign the token. As we’ll see in section 7.3.4, we can choose from various types of algorithms when we sign a JWT.

kid: the ID of the key or secret used to sign the token.

As you’ll learn in chapter 8, we validate token signatures by selecting the right algorithm and signing key, and we collect this information from the alg and kid fields in the JWT header.

Listing 7.1 Example of JSON Object Signing and Encryption (JOSE) header

{
  "typ": "JWT",
  "alg": "RS256",
  "kid": "080E8CtRFAnnLlgK3dk8Y"
}

The next element in the JWT is the payload. The payload contains claims about the token’s subject, which can be a user or a non-human identity such as a service. In the case of ID tokens, the payload contains identifying information about the user, while access token payloads contain claims about the right of a user to access an API. We’ll talk more about ID tokens in section 7.6 and focus on access tokens here.

An access token payload contains all the necessary claims for a user to prove they have access to an API. RFC 7519 [“JSON Web Token (JWT)” by M. Jones, J. Bradley, and N. Sakimura, May 2015, https://datatracker.ietf.org/doc/html/rfc7519] defines seven reserved or registered claims:

iss (issuer): identifies the authorization server that issued the token.

sub (subject): identifies the subject the token belongs to. The subject identifies a human identity, such as a user, or a non-human identity, such as a machine-to-machine client. All the claims in the token apply exclusively to this subject.

aud (audience): identifies the API the token gives access to.

exp (expiration): when the token expires.

nbf (not before): a timestamp before which the token isn’t valid.

iat (issued at time): when the token was issued.

jti (JWT ID): a unique identifier for the token.

Registered claims aren’t mandatory, which means we are free to issue tokens without them. However, this set of claims is well understood in the industry, and it is best practice to include them in our tokens since it improves their interoperability, making them easier to use in other systems right out of the box. The JWT ecosystem is built around RFC 7519, and that means standard JWT libraries know how to validate tokens with registered claims. Some popular API products, like API gateways (see section 9.1), support JWT authentication and can handle tokens with registered claims too. Listing 7.2 shows an example of JWT payload using all the reserved claims.

Listing 7.2 Example JWT payload with reserved claims

{
  "iss": "https://auth.apithreats.com/",
  "sub": "b896ff3a",
  "aud": "https://apithreats.com",
  "exp": "1823408279",
  "nbf": "1823321879",
  "iat": "1823318279",
  "jti": "873137f26e8d"
}

In addition to the reserved claims, we can include custom claims in JWTs. Custom claims allow us to be more specific about the access rights of the user, for example by specifying their user role or group, their organization, and similar attributes. A common use case is including the user role in the token for role-based access controls, which we’ll learn more about in section 7.8.

When designing custom claims for your tokens, consider carefully what type of information you’re leaking through them, and make sure you don’t include sensitive user data, like personal names and emails. The user identifier is already available under the standard sub claim, and if you need additional information about the user, you can obtain that data by querying the identity service directly. For example, as we’ll see in section 7.6, most OpenID providers expose a UserInfo endpoint where we can query user details.

What exactly is the risk of including sensitive data in access tokens? As illustrated in figure 7.3, access tokens are included in requests to the API. If a user connects to an API using an insecure connection on a public network, they may become vulnerable to a man-in-the-middle attack or eavesdropping, making their data accessible to threat actors. Access tokens can also be hijacked if our website has misconfigured cross-origin resource sharing (CORS) headers, is vulnerable to headers or HTML injection, and other factors.

Figure 7.3 If a user connects to a website using an insecure connection on a public network, threat actors may be able to hijack their access tokens by capturing network packets using a packet analyzer such as Wireshark.

The final element of a JWT is the signature. The process for signing tokens is described in RFC 7515 [“JSON Web Signature (JWS)”, by M. Jones, J. Bradley, and N. Sakimura, May 2015, https://www.rfc-editor.org/rfc/rfc7515.html]. The signature is the result of applying the signing algorithm over the JWT’s header and payload, and hence it conveys the integrity and authenticity of the token. We produce the JWT signature by applying a signing algorithm on base64url encoded value of the header and the payload, separated by a dot. For example, the token header and payload in listing 7.3 results in the following string when we base64url encode and join them by a dot:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2FwaXRocmVhdHMuZX
[CA]UuYXV0aDAuY29tLyIsInN1YiI6ImI4OTZmZjNhIiwiYXVkIjoiaHR0cHM6Ly9hcGl0aHJlY
[CA]XRzLmNvbSIsImV4cCI6IjE4MjM0MDgyNzkiLCJuYmYiOiIxODIzMzIxODc5IiwiaWF0Ijoi
[CA]MTgyMzMxODI3OSIsImp0aSI6Ijg3MzEzN2YyNmU4ZCJ9

This becomes the input to the signing function, also known as the JWS signing input. The alg parameter in listing 7.3 is set to HS256, which means the token is signed with a hashing function (HMAC-SHA256). HMAC uses a secret key to produce a message authentication code, which we use as the token’s signature. Using password as the secret key, we obtain the following signature: -WISLJq1jkQw1vY4IVaAKxLAL_hxoFnT9oLfBpgPvzA. In chapter 8, you’ll learn to how produce tokens with practical coding examples.

Listing 7.3 Example JWT payload with reserved claims

# Header
{
  "typ": "JWT",
  "alg": "HS256"
}
# Payload
{
  "iss": "https://auth.apithreats.com/",
  "sub": "b896ff3a",
  "aud": "https://apithreats.com",
  "exp": "1823408279",
  "nbf": "1823321879",
  "iat": "1823318279",
  "jti": "873137f26e8d"
}

We can sign JWTs using a wide range of algorithms. The full list of signing algorithms and how they work is documented in RFC 7518 [“JSON Web Algorithms (JWA)” by M. Jones, May 2015, https://www.rfc-editor.org/rfc/rfc7518.html]. IANA also keeps a list of the available choices (https://www.iana.org/assignments/jose/jose.xhtml, section “JSON Web Signature and Encryption Algorithms”), with recommendations on which algorithms are required for libraries implementing JOSE.

“Help! I don’t want to expose my API’s JWT claims!”

As you’ve learned in this section, JWTs are base64url encoded, which means anyone who gets their hands on a token can inspect its contents (see section 8.2 to learn how). But what if the claims are so sensitive that you don’t want anyone, including their legitimate owners, to be able to inspect them? In that case, we encrypt the token.

RFC 7516 (“JSON Web Encryption (JWE),” by M. Jones and J. Hildebrand, May 2015 https://www.rfc-editor.org/rfc/rfc7516.html) describes a standard method for encrypting sensitive content, including access tokens with sensitive information. In cryptography, we call the message to be encrypted the plaintext. JWE uses authenticated encryption, which allows us to encrypt a message and protect the integrity of additional data. Additional authenticated data (AAD) is data that don’t want to encrypt, but we want to make sure it cannot be modified by a malicious party. Authenticated encryption is an encryption technique that produces a ciphertext (the encrypted plaintext) and an authentication tag. We use the authentication tag to validate the integrity of the ciphertext and the additional data.

JWEs allow us to transmit encrypted data in a structured way that the receiving party can understand, and they contain the following elements:

JOSE protected header: it describes the encryption applied to the plaintext and, optionally, other properties of the JWE. The algorithms included in the header must be chosen from the list of algorithms listed in sections 4.1 and 5.1 of RFC 7518 (“JSON Web Algorithms (JWA),” by M. Jones, May 2015, https://datatracker.ietf.org/doc/html/rfc7518). The JOSE header represents the additional data that we use in the authenticated encryption process and it is integrity-protected by the authentication tag.

Encrypted key: the content encryption key used to encrypt the plaintenxt, encrypted.

Initialization vector: a random value used to set the initial state of the encryption algorithm.

Ciphertext: the result of encrypting the plaintext.

Authentication tag: a value produced by the encryption algorithm to prove the integrity of the JWE, i.e. to show that neither the ciphertext nor the plaintext were modified.

The JWE’s JOSE header contains at least two fields:

"alg": it identifies the algorithm used to encrypt the cryptographic encryption key.

"enc": it identifies the algorithm used to produce the ciphertext and the authentication tag.

For example, a JWE JOSE header may look like this:

{"alg":"RSA-OAEP","enc":"A256GCM"}

This header indicates that the encryption key is encrypted using the RSA-OAEP (RSA with optimal asymmetric encryption padding) algorithm and that the ciphertext and the authentication tag were produced using the A256GCM (AES-256 with Galois/Counter Mode) algorithm.

RFC 7516 describes two methods for JWE serialization, i.e. for putting all the above information together: compact and JSON serialization. In compact serialization, we base64url encode every element of the JWE and we concatenate them using dots as separators:

Base64url(JOSE header) || '.' ||
Base64url (encrypted key)_|| '.' ||
Base64url (initialization vector) || '.' ||
Base64url (ciphertext) || '.' ||
Base64url (authentication tag)

If we use JSON serialization, the whole JWE is represented as a JSON data structure like the following:

{
  "protected":"<the integrity-protected JOSE header>",
  "encrypted_key":"<the encrypted key>",
  "iv":"<the initialization vector>",
  "ciphertext":"<the ciphertext>",
  "tag":"<the authentication tag>"
}

The JSON is then base64url encoded.

To learn more about JWE, check out Scott Brady’s “Understanding JSON Web Encryption (JWE),” August 2022 (https://www.scottbrady.io/jose/json-web-encryption).

The most widely supported signing algorithms are HS256 and RS256. HS256 stands for HMAC with SHA-256. HMAC is a message authentication code (HMAC) algorithm that uses a hash function and a secret key. HS256 uses SHA-256 as the hashing algorithm, and it produces a code (the C in HMAC) that verifies the integrity of the message, i.e. it proves that the message is authentic and has not been tampered. The strength of the signature is a function of the strength of the secret key and the underlying hash function. Short secret keys make the signature vulnerable to brute force attacks.

We use the output of the HS256 algorithm to sign JWTs. HS256 is a symmetric signing algorithm, which means we use the same secret key to produce the signature and to validate it. This increases the risk of leakage since the key is available in multiple places and is always necessary for validation. For all those reasons, it’s generally preferred the use of asymmetric encryption algorithms, such as RS256.

RS256 uses the Rivest–Shamir–Adleman (RSA) key-pair cryptosystem’s signature scheme to sign tokens using a private key, and a public key to verify the signature. Specifically, RS256 refers to signature scheme RSASSA-PKCS1-v1_5, as defined in RFC 3447 [“Public-Key Cryptography Standards (PKCS) #1: RSA Cryptography Specifications Version 2.1”, by J. Jonsson and B. Kaliski, Feb 2003, https://www.rfc-editor.org/rfc/rfc3447], using SHA-256 to hash the message. RS256 is an asymmetric signing algorithm since it uses different keys for signing and verification.

Despite its popularity, RS256 has some well-known vulnerabilities. For example, in 2006, Daniel Bleichenbacher demonstrated that incorrect implementations of RSASSA-PKCS1-v1_5 were vulnerable to signature forgery attacks, and sadly, many implementations have such vulnerabilities (for more on this see David Wong’s book, Real-World Cryptography, section 7.3.1, published by Manning in 2021). For that reason, RFC 8725 (“JSON Web Token Best Current Practices” by Y. Sheffer, D. Hardt, and M. Jones, Feb 2020, https://datatracker.ietf.org/doc/html/rfc8725) recommends avoiding RS256 and using instead improved versions of the algorithm such as RSAES-OAEP (optimal asymmetric encryption padding), and PS256 (probabilistic signature scheme) or elliptic curve digital signature algorithm (ECDSA) signatures. You can learn more about these algorithms with David Wong, Real-World Cryptography, sections 6.3.3, 7.3.2, and 7.3.3, and RFCs 7518 (https://www.rfc-editor.org/rfc/rfc7518.html) and 3447 (https://www.rfc-editor.org/rfc/rfc3447).

7.3 Understanding Open Authorization

Open Authorization (OAuth) is a protocol for access delegation. OAuth allows us to share information about ourselves with other applications without having to share our passwords. For example, if we’re filling in our details on a job portal website and we want to import our profile from LinkedIn, how does the job portal access our information? As shown in figure 7.4, one option is to share our LinkedIn credentials with the job portal to let it access LinkedIn on our behalf and retrieve our profile. This is dangerous, though, since it gives the job portal unfettered access to our LinkedIn account.

Figure 7.4 To give the job portal access to her employment history, Susan shares her LinkedIn credentials. This is risky, since it gives the job portal full access to Susan’s profile on LinkedIn.

OAuth was developed to address this problem. To grant access to a third-party application like a job portal, to our data on LinkedIn, we want to ensure that access is

limited to certain data and operations, for instance, read our employment history

valid only for a certain period of time

OAuth uses three concepts to implement such restrictions:

Authorization flow: the process we use to authorize an application to access our data on another website. There are different flows depending on the type of application we are authorizing.

Scopes: when requesting access to our data, third-party applications must specify what data and operations they want to access. We call those access constraints scopes.

Access tokens: this is what gives the third-party application access to our data on another website. Access tokens are valid for a limited period of time and their access is constrained to the scopes requested during the authorization flow.

To put this all together, OAuth uses the concept of authorization flows. As illustrated in figure 7.5, the authorization flow is the process by which a third-party application gains access to our data on another website. There are four important elements in every authorization flow:

Authorization server: the server that manages user identities and issues their access tokens.

Resource server: the server that stores user data and controls access to it.

Resource owner: the user who owns data in the resource server.

Client: the application trying to access data in the resource server.

Figure 7.5 Open Authorization describes flows through which users (resource owner) can authorize applications (client) to access their data in an API (resource server). Every flow begins with an authorization request, followed by user login and consent, and finally an access token grant.

We distinguish between confidential clients and public clients. Confidential clients are applications that can store credentials securely and therefore can use them to authenticate their requests with the authorization server. Public clients are clients whose source code is exposed to everybody, such as browser-based and mobile applications. Since the source code is publicly visible, public clients can’t store credentials securely and therefore can’t authenticate their requests with the authorization server.

The OAuth specification describes various authorization flows for different scenarios. For example, we have an authorization flow to authorize machine-to-machine integrations, like integrations with third-party APIs, while another flow allows us to authorize access from a browser or mobile application. However, all the flows follow the same structure. As illustrated in figure 7.5, an OAuth flow consists of four steps:

Authorization request: the initial request in the authorization flow, in which the client requests access to an API. Typically, it contains details such as the client ID and the access scopes required to access the API.The authorization request prompts the user to log in with the identity provider.

User consent: after successful login, the user gives consent for the client application to obtain an access token.

Issue access token: after user consent, the authorization server issues the access token.

Access resources: once the client has obtained the access token, it can access user data in the resource server.

At the beginning of this section, I mentioned that OAuth was developed to address the problem of sharing data with third-party applications. So how is this relevant in the context of our own APIs? In most cases, OAuth comes to life through OpenID Connect, a popular authentication protocol that we’ll explore in more detail in section 7.6. But first, let’s break down the OAuth flows to understand in detail how they work and when to use them!

7.4 Understanding OAuth flows

The OAuth specification acknowledges that there are different scenarios in which we need to authorize access to an API. Each scenario imposes its own constraints, and OAuth describes different flows tailored to the needs of each situation. The flows supported by the OAuth specification have changed over time. As of OAuth 2.1, this is the current list of supported flows:

Authorization code flow

Client credentials flow

The device authorization grant

Refresh token flow

OAuth 2.1 also deprecates two flows from the OAuth 2.0 specification:

Implicit flow

Resource owner password flow

The resource owner password flow involves handing user credentials to the third-party application. In the example of the job portal accessing our LinkedIn profile from the previous section, this means handing our LinkedIn username and password to the job portal. This is unsafe since it gives the third-party application full control over our LinkedIn account.

The implicit flow was introduced in OAuth 2.0 to support authorization for single-page and mobile applications. The biggest flaw in this flow is that it delivers the access token through the URL, hence exposing it to unauthorized parties who can use it to hijack user permissions.

7.4.1 Authorization code flow

The authorization code flow consists of the exchange of an authorization code for an access token. This flow is optimized for confidential clients such as web servers, where we can store confidential configuration. As illustrated in figure 7.6, this flow uses redirections; for instance, it begins by redirecting the resource owner to the authorization server, and ends by redirecting them back to the application. To carry out such redirections, the client must be able to interact with the resource owner’s user-agent (aka the browser). Therefore, this flow works well for web applications.

To illustrate how this flow works, let’s go back to the example of the job portal that needs to access our LinkedIn profile. As illustrated in figure 7.6, the job portal (aka the client) initiates the authorization process by producing an authorization URL and redirecting the resource owner to it (the authorization request). The authorization request URL takes the resource owner to LinkedIn (aka the authorization server), where they will prove their identity and grant access to their data.

Figure 7.6 In the authorization code flow, a confidential client, such as a web server, uses a client ID and a client secret to obtain the authorization code.

The authorization request URL contains a client ID (client_id) and a callback or redirect URI (redirect_uri). The URL also contains a response_type parameter, which is used by clients to indicate which authorization flow they wish to use. For the authorization code flow, the response_type’s value is code. A fully formed authorization request URL or the authorization code flow looks like this:

https://auth.example.com/authorize?response_type=code
[CA]&client_id=s6BhdRkqt3 &redirect_uri=https://example.com

After the resource owner grants the client access to their data, the authorization server issues an authorization code and sends the user back to the client application, using the address indicated in the redirect URI, and injecting the authorization code in the URL. At this point, the client application grabs the authorization code from the URL and exchanges it for an access token with the authorization server.

The authorization code flow uses a client ID and a client secret to keep the whole transaction safe. The client secret is highly sensitive and must never be exposed to external users. The client ID is a unique identifier for the client, the job portal in our example. When the authorization server issues the authorization code, it binds the code to the client ID from the authorization URL. When the client exchanges the authorization code for an access token, it must provide the client secret to prove that the request comes from a legitimate source. It’s best practice to make authorization codes for one-time use only.

DEFINITION

OAuth CSRF attacks are attacks in which a malicious actor steal a user’s access token by exploiting the flow’s redirection feature. The threat actor uses a malicious link that initiates the authorization flow on the user’s behalf. If the user has an active session with the authorization server, the login step will be skipped and the user will be immediately redirected to the redirect URL indicated in the malicious link with the authorization code, which the threat actor’s server then exchanges for an access token. This type of exploit is possible when authorization servers are vulnerable to open redirect attacks, which occur when they don’t constrain the values allowed in the redirect_uri parameter.

For additional security, the authorization server may require including a state parameter in the authorization request URL (the first step). The state parameter is an opaque, non-guessable value that is bound to the client’s authorization request, and it must be included in every request and response of the flow until the final exchange for the access token. This parameter protects users from access token hijacking via cross-site request forgery (CSRF) attacks. A fully formed URL with the state parameter looks like this:

https://auth.example.com/authorize?response_type=code
[CA]&client_id=s6BhdRkqt3 &redirect_uri=https://example.com

OAuth 2.1 also requires using the authorization code with an additional security mechanism known as proof of key exchange, which helps to prevent authorization code hijacking attacks. In the next section, we describe how this works.

7.4.2 Protecting authorization requests with proof of key exchange

The final step of the authorization code flow is the exchange of the authorization code for an access token. If a threat actor lands their hands on the authorization code, using a man-in-the-middle attack, a CSRF attack, or another strategy that allows them to intercept authorization server responses, nothing prevents them from exchanging the authorization code for an access token before the legitimate user does, and therefore, impersonate the user and gain access to their data. How do we prevent such attacks? In other words, how can we improve the flow so that the authorization server can distinguish between legitimate and spoofed authorization requests? The answer is proof of key exchange.

RFC 7636 (“Proof Key for Code Exchange by OAuth Public Clients” by N. Sakimura, J. Bradley, and N. Agarwal, September 2015 [https://www.rfc-editor.org/rfc/rfc7636]) describes a security mechanism known as proof of key exchange (PKCE, read “pixi”) that helps to prevent authorization code interception attacks by introducing two new request parameters: a code challenge and a code verifier. Let’s see how it works. Figure 7.7 illustrates the steps involved in the PKCE flow.

Figure 7.7 Proof of key exchange (PKCE) prevents authorization code interception attacks by requiring the client to prove ownership of the authorization code by supplying a valid code verifier.

To initiate the authorization flow, the client application produces a code verifier and a code challenge. The code verifier is a high-entropy random value with a minimum of 43 characters and a maximum of 128. The code verifier must be unique in every authorization request. The client application hashes the code verifier using SHA-256 hashing (S256) to produce the code challenge, and then base64 encodes the output for safe transmission. The base64 encoding must not include padding.

As shown in figure 7.7, the client redirects the resource owner to the authorization server, including the code challenge and a callback URL in the authorization request step. The client includes the code challenge in the code_challenge URL query parameter, and the hashing method using the code_challenge_method URL query parameter.

Once the resource owner proves their identity and grants access to the client application, the authorization server redirects the resource owner back to the client application using the callback URL, including an authorization code in the URL. Finally, the client exchanges the authorization code for an access token, including the code verifier in the request. The authorization server hashes the code verifier using the method indicated by the client in the authorization request and compares its value with the code challenge. If the outcome is successful, it responds to the client with an access token.

7.4.3 Client credentials flow

The client credentials flow is designed for machine-to-machine communication. This flow is suitable for integrations where the API consumer is a non-human identity. In such situations, the API client often has constraints such as being unable to use a browser to perform the complex series of steps described in the authorization code flow. Here are some examples of use cases for the client credentials flow:

Automations that require an API call, like needing to call an API during a continuous integration process.

System integrations with third-party APIs to consume functionality such as sending emails, geolocation, processing payments, and so on.

Integrations between internal services in distributed architectures, such as microservices.

In all these cases, we need to be able to authenticate every request. However, if you look at the steps involved in the authorization code, they certainly don’t look suitable for machine-to-machine integrations where we can’t expect the API client to be able to interact with a browser as a human would. The question is, how do we authenticate API clients in these scenarios? This is where the client credentials flow comes in. As shown in figure 7.8, the client credentials flow is a much simpler flow that involves the exchange of credentials directly for an access token.

Figure 7.8 In the client credentials flow, a server-side application exchanges credentials directly for an access token. If needed, the resource owner gives consent upon registering the client with the authorization server.

As illustrated in figure 7.8, the client credentials flow is different from other flows in that it doesn’t require a browser and there’s no consent step. If a client acts on behalf of a resource owner, consent is given upon registering the client application with the authorization server. For instance, imagine you want to use a service to automate publishing posts on LinkedIn so that you don’t have to spend so much time on social media and can focus on your work. To make this work, you register an application to publish on your behalf on LinkedIn with a given schedule. The application will need to obtain authorization every time it publishes on LinkedIn. However, it won’t be able to request your consent on every authorization request. To satisfy this constraint, we give consent to the application when we register it, stating that we allow the application to act on our behalf whenever it interacts with LinkedIn.

In service-to-service integrations, there’s typically no consent involved. When we register a service client with the authorization server, we define its permissions and boundaries, and this ensures that each service doesn’t get access to data and operations that don’t belong in its domain.

7.4.4 Device authorization flow

Have you ever signed in to Netflix or YouTube from a smart TV? If so, you’ve used the device authorization flow. The device authorization flow is designed for input-constrained devices, such as smart TVs, and for agentless devices. Input-constrained means that it’s difficult or unreasonable to ask the user to type in their credentials. For example, it’d be very challenging to type in a 50-plus-strong character password using a TV remote controller. Agentless devices are devices that can’t open a browser to follow the typical OAuth flow, and this includes IoT devices, printers, and more.

For such situations, we use the device authorization flow. As illustrated in figure 7.9, the client application, such as a smart TV, beings by requesting access from the authorization server. This authorization request includes the client identifier. The authorization server responds by issuing a device code, an end-user code, and a verification URL. The device then asks the user to visit the verification URL, where they sign in and input the end-user code to authorize the device’s request. In the meantime, the device polls the authorization server using the device code and the client identifier to find out if the authorization request is completed. Once the user has granted access, the authorization server responds to the device with an access token.

Figure 7.9 The device authorization flow is designed for input-constrained devices such as smart TVs. In this case, the resource owner logs in and gives consent through an external application, using a user code and a verification URL provided by the device.

To learn more about device authorization flow, check out the RFC 8628, “OAuth 2.0 Device Authorization Grant” by W. Denniss, J. Bradley, M. Jones, H. Tschofenig (https://datatracker.ietf.org/doc/html/rfc8628).

7.4.5 Refresh token flow

The refresh token flow is the process by which clients renew their access. Access tokens have an expiration date, typically within 1 to 24 hours of issuing them, and when they expire, clients must renew their access using a refresh token. Refresh tokens are opaque to the client and they typically consist of a random sequence of characters. As shown in figure 7.10, refresh tokens are issued together with access tokens, and upon exchange, the client receives a new access token and a new refresh token.

Figure 7.10 The refresh token flow allows clients to renew their access token without having to ask the resource owner to log in and give consent all over again.

A special security consideration in the refresh token flow is how to prevent threat actors from using the refresh token in case it’s leaked. The OAuth 2.1 specification says that authorization servers must bind the refresh token to the client (https://www.ietf.org/archive/id/draft-ietf-oauth-v2-1-11.html#name-refresh-token-grant). With confidential clients, such as the type of client used in the authorization code flow (for example, a web server), this is typically accomplished by using the client ID plus the client secret. In the PKCE flow, we don’t have client identifiers, and in that case, the OAuth 2.1 specification recommends using sender-constrained tokens with methods such as demonstrating proof of possession and mutual TLS, which we describe in the next section.

Refresh tokens should be one-time use only to prevent their reuse by malicious actors in case they intercept them, and they should be valid for a limited period of time. There is no typical length of time for which refresh tokens are valid. Make sure they are available for a reasonable period of time after access token expiration, and tailor their expiration to the security needs of your application. Once a refresh token has expired, the client needs to authenticate again to obtain new access and refresh tokens.

7.5 Sender-constrained tokens

One of the most critical security concerns in web security is how to ensure that the right access credential is sent by the right user. When a user logs in to our website, we issue an access token for them to access our APIs. But what if the token is leaked and falls into the hands of a malicious actor? Access tokens carry the user permissions to access their data, so without being able to verify that the legitimate owner of the access token is accessing our API, we’re putting our users at risk. This is where sender-constrained tokens come in.

Sender-constrained tokens are tokens bound to a specific sender (aka client application). The OAuth 2.1 specification describes two methods we can use to bind tokens to a user: demonstrating proof of possession and mutual TLS (https://www.ietf.org/archive/id/draft-ietf-oauth-v2-1-11.html#name-sender-constrained-access-t). Let’s see how it works.

7.5.1 Mutual TLS for certificate-bound tokens

Mutual TLS (mTLS) is an authentication method built on top of the Transport Layer Security (TLS) protocol. TLS is the method modern websites use to secure and encrypt communication between a client (such as a browser) and a server. Web servers use asymmetric encryption with TLS certificates to secure the connection and prove their identity. As shown in figure 7.11, when our browser connects to a website, it downloads the website’s public certificate and uses that information to verify the server’s identity during the TLS handshake.

Figure 7.11 Transport layer security (TLS) describes the protocol clients (for example, browsers) and web servers follow to prove the server’s identity and secure the connection. We call this process TLS handshake. When the browser connects to the server, it downloads the server’s public key, verifies its authenticity, and uses it to encrypt the requests.

Mutual TLS involves authentication from the client too. As illustrated in figure 7.12, mTLS requires the client to prove their identity by sending a public certificate to the server during the TLS handshake. Depending on the arrangement, this certificate could be issued by the server or by the client itself. The client proves ownership of the certificate by signing a message using the certificate’s private key during the TLS handshake’s CertificateVerify step from figure 7.12.

Figure 7.12 Mutual TLS requires the client (for example, a browser) to provide their own certificate to verify their identity.

In the context of Open Authorization, we use mTLS to bind a token to a specific client certificate. When establishing a connection with an authorization server, the server captures the details of the client’s certificate. With this information, the authorization server produces a thumbprint of the client’s public certificate, and includes it in the access token using the cnf claim, as illustrated in listing 7.4.

DEFINITION

A certificate thumbprint, of fingerprint, is a unique identifier for digital certificates which we obtain by calculating the SHA-1 digest of the certificate’s DER representation. DER is a standard binary encoding format for representing data structures, including certificates. To include the certificate thumbprint in a JWT, we base64url encode its value.

Listing 7.4 JWT with certificate thumbprint confirmation method

# Header
{
  "typ":"JWT",
  "alg":"RS256",
}
.
# Payload
{
  "iss": "https://auth.apithreats.com",
  "sub": " 8bda8694",
  "iat": 1816158609,
  "exp": 1816165809,
  "nbf": 1816158609,
  "cnf":{
    "x5t#S256": "bwcK0esc3ACC3DB2Y5_lESsXE8o9ltc05O89jdN-dg2"
  }
}.
SIGNATURE

The cnf claim was introduced in RFC 7800 as a method for including proof of possession of key information within a JWT. RFC 8705 builds on top of that specification by introducing specific semantics to prove possession of a X.509 certificate through a x5t#S256 member within the cnf claim. The x5t#S256 member represents the base64url encoded value of the certificate’s thumbprint.

Note

Mutual TLS for certificate-bound access tokens is documented in RFC 8705, “OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens”, by B. Campbell, J. Bradley, N. Sakimura, and T. Lodderstedt (https://www.rfc-editor.org/rfc/rfc8705.html). RFC 8705 builds on top of the TLS specifications 1.2 and 1.3. TLS 1.2 is documented in RFC 5246 (“The Transport Layer Security (TLS) Protocol Version 1.2” by T. Dierks and E. Rescorla, Aug 2008, https://datatracker.ietf.org/doc/html/rfc5246) and TLS 1.3 in RFC 8446 (“The Transport Layer Security (TLS) Protocol Version 1.3” by E. Rescorla, Aug 2018, https://datatracker.ietf.org/doc/html/rfc8446). RFC 8705 also builds in top of RFC 7800 (“Proof-of-Possession Key Semantics for JSON Web Tokens (JWTS),” by M. Jones, J. Bradley, and H. Tschofenig, April 2016, http://datatracker.ietf.org/doc/html/rfc7800) by leveraging the proof-of-possession semantics of the cnf claim.

When the client sends a request to the resource server, the server must capture details of the client’s certificate during the TLS handshake. Then it uses that information to verify that it matches the certificate’s thumbprint in the access token, hence verifying that the token belongs to the client.

Despite its benefits, mTLS for certificate-bound tokens is difficult to implement. Servers must be able to capture details from the client’s certificate at the protocol level and make them available at the application level, where we validate access tokens. Since the TLS handshake happens when a client establishes a connection with a server, this information must be captured before knowing what type of action the user wishes to perform. An easier alternative to bind client certificates to access tokens was proposed in a recent RFC, as explained in the next section.

7.5.2 Demonstrating proof of possession

Demonstrating proof of possession (DPoP) is a novel method for binding access tokens to a sender using a sender-generated cryptographic private key. The sender proves possession of the key and hence entitlement to the access token by including a special header signed with a cryptographic key generated by the sender. Figure 7.13 illustrates how DPoP works.

Figure 7.13 Demonstrating proof of possession (DPoP) binds the access token to the public key thumbprint of the client’s certificate. The client proves ownership of the certificate by including a newly signed DPoP token in every request.

Note

Demonstrating proof of possession (DPoP) is documented in RFC 9449, “OAuth 2.0 Demonstrating Proof of Possession (DPoP)”, by D. Fett, B. Campbell, J. Bradley, L. Lodderstedt, M. Jones, and D. Waite (https://datatracker.ietf.org/doc/html/rfc9449).

As illustrated in figure 7.13, everything begins with the sender choosing a signing algorithm and generating the corresponding signing key pair. DPoP uses asymmetrical signature algorithms such as RS256. Then the client sends the authorization request to the authorization server, including in the dpop_jkt parameter the JWK SHA-256 thumbprint of the DPoP public key, base64url encoded. For example:

GET /authorize?response_type=code&state=xyz
[CA]&redirect_uri= https%3A%2F%2Fapithreats.com%2Fdocs
[CA]&code_challenge_method=S256
[CA]&dpop_jkt=NzbLsXh8uDCcd-6MNwXF4W_7noWXFZAfHkxZsRGC9Xs

The authorization server then issues an authorization code bound to the public key’s thumbprint. To exchange the authorization code for an access token, the sender must prove possession of the private key corresponding to the public key’s thumbprint sent earlier. We do this by creating a special DPoP JSON Web Token (JWT) and including it in the DPoP request header. As you can see in listing 7.5, a DPoP JWT contains a header, a payload, and a signature.

Listing 7.5 Structure of a DPoP JWT

# Header
{
  "typ":"dpop+jwt",
  "alg":"ES256",
  "jwk": {
    "kty":"EC",
    "x":"l8tFrhx-34tV3hRICRDY9zCkDlpBhF42UQUfWVAWBFs",
    "y":"9VE4jf_Ok_o64zbTTlcuNJajHmt6v9TDVrU0CdvGRDA",
    "crv":"P-256"
  }
}
.
# Payload
{
  "jti":" 2e3e5c23",
  "htm":"POST",
  "htu":"https://apithreats.com",
  "iat": 1816165809,
  "exp": 1816169409,
  "nbf": 1816165809,
  "ath":"fUHyO2r2Z3DZ53EsNrWBb0xWXoaNy59IiKCAqksmQEo"
}
.
SIGNATURE

The DPoP JWT header contains a typ field describing the token as a DPoP token, an alg field indicating the algorithm used to sign the token, and a jwk object that represents the public key. The contents of the jwk object depend on the type signing algorithm. In this case, it describes a P-256 elliptic curve key with coordinates x and y. The x and y values are the base64url encoding of the octet representation of the numerical values. The key thumbprint is the base64url encoding of the SHA-256 hashing of the octet representation of the jwk object.

The DPoP payload contains a unique token ID (jti), the HTTP method (htm) and URI (htu) of the request in which this JWT is included, the time when the JWT was issued (iat), and a hash of the access token (ath). The sender signs the token with the private key previously generated.

Upon receiving the token request, the authorization server verifies that the DPoP belongs to the sender by validating its signature against the JWK thumbprint from the authorization request. If verification is successful, the authorization server issues an access token and a refresh token bound to the sender’s signing key. To bind an access token to a DPoP key, RFC 9449 introduces a new claim in the token’s payload called cnf, which represents the JWT confirmation method. For DPoP tokens, the confirmation method is signature validation using the public key’s thumbprint. Listing 7.6 shows an example of an access token bound to a DPoP key with the cnf field.

Listing 7.6 JWT access token bound to a DPoP key

{
  "sub":"079e1799",
  "iss":"https://auth.apithreats.com",
  "nbf":1562262611,
  "exp":1562266216,
  "cnf":
  {
    "jkt":"0ZcOCORZNYy-DWpqq30jZyJGHTN0d2HglBV3uiguA4I"
  }
}

Once the sender has received the access token, they can access resource server. The sender must include the DPoP request header in every request to prove possession of the token, and the resource API must validate the DPoP token’s validity using the thumbprint confirmation method.

DPoP JWTs are for one-time use only, and every new request to the resource server requires the sender to produce a new DPoP JWT. As we saw in listing 7.6, the DPoP JWT contains information about the request it’s bound to, including the request URI and HTTP method. They also contain a unique identifier under the jti field. All this helps to prevent replay of DPoP JWT headers by threat actors if the token gets leaked.

7.6 Understanding OpenID Connect

OpenID Connect (OIDC) is an authentication protocol that allows users to bring their identity from one website to another. For example, when you log in to a new website using your Gmail or GitHub accounts, you’re using OIDC – you’re using your Gmail or GitHub identity to register with and log into a new website. OIDC gives identity providers (IDP) a standard way to expose and manage their authentication flows. If you are looking to outsource authentication to IDPs like Auth0, Okta, Microsoft Entra, Curity, or Authlete, you’ll be using OIDC to integrate with them. Let’s see how it works!

OIDC builds on top of the OAuth 2.0 specification, and it uses OAuth flows to authenticate user. The outcome of a successful OIDC flow is an ID token, and optionally, an access token for the OpenID provider API. Every authentication flow in OIDC includes the following parties:

OpenID provider (OP) or identity provider (IDP): the server that manages user identities and authenticates them. This is the equivalent of the authorization server in OAuth.

Relying party (RP): the application that outsources authentication to an OpenID or identity provider.

User: the user whose identity the RP is trying to verify. This is the equivalent of the resource owner in OAuth.

Client: the application requesting ID and access tokens on behalf of the user. Clients must be registered with the identity provider to be able to obtain user information. For example, to include Gmail authentication in our website, we must register a client with Google Identity.

As illustrated in figure 7.14, OIDC authentication flows include the following steps:

Authentication request: the client puts together a URL that redirects the user to the identity provider where they authenticate. The authentication contains details like the client ID and it includes openid in its list of scopes.

User consent: after successful login, the user gives consent for the client application to access their data.

Issue ID token and optionally an access token: after user consent, the identity provider issues ID token that contains information about the user, and optionally, an access token that gives access to a protected UserInfo endpoint.

Client requests user data from the UserInfo endpoint: UserInfo is a protected resource in OpenID servers that contains user data. Clients can use this endpoint to obtain additional information about the user.

Figure 7.14 OpenID Connect allows applications to use OpenID providers to manage their user identities securely. In this case, the application redirects the user to the OIDC server to log in and give consent, and it obtains an ID token and an access token in return.

In many cases, the identity provider works also as an authorization server and can issue access tokens. In such cases, we can combine OIDC with OAuth to authenticate users and obtain their access tokens at the same time. This is a common use-case when we outsource authentication to identity providers like Auth0, Okta, or Microsoft Entra.

ID tokens follow the JWT specification (RFC 7519, https://datatracker.ietf.org/doc/html/rfc7519) and contain a header, a payload, and a signature as we saw in section 7.2.1. ID token payloads contain all the registered claims of RFC 7519 (see section 7.2.1) plus claims about the authentication process and the user identity. The following claims describe the authentication process and are all optional:

auth_time: a UTC timestamp representing the time when the user logged in.

acr: a URI that identifies the authentication context class reference used to authenticate the user. Authentication context class refers to the methods used during authentication and the level of assurance that clients and relaying parties can place on the strength of the authentication process, and whether it’s adequate for the level of sensitivity of the resources being accessed. The level of assurance is defined following the guidelines from ISO/IEC 29115 (https://www.iso.org/standard/45138.html), and the authentication context class URI is defined following RFC 6711’s guidelines (“An IANA Registry for Level of Assurance (LoA) Profiles” by L. Johansson, Aug 2012, https://www.rfc-editor.org/rfc/rfc6711.html). OpenID’s “Provider Authentication Policy Extension” (PAPE, https://openid.net/specs/openid-provider-authentication-policy-extension-1_0.html) specification provides some examples of ACRs, such as http://schemas.openid.net/pape/policies/2007/06/phishing-resistant for phishing-resistant authentication (note that this URL is for illustration only and won’t work if you try to open it).

amr: an array of strings describing the authentication methods employed to authenticate the user, using the reference values in the “Authentication Method Reference Values” specification (Feb 2017, by M. Jones, P. Hunt, and A. Nadalin, https://datatracker.ietf.org/doc/html/draft-ietf-oauth-amr-values-06). For example, the reference value for password-based authentication is pwd and for one-time password is otp. Check out the specification for the full list of values.

azp: the authorized party the ID token was issued for, aka the client ID.

In addition to claims about the authentication process, ID tokens also include claims about the user identity, such as name, profile, picture, email, website, phone_number, and so on. You can check the full list of user identity claims in the “OpenID Connect Core 1.0” document (by N. Sakimura, J. Bradley, M. Jones, B. de Medeiros, and C. Mortimore, Dec 2023, https://openid.net/specs/openid-connect-core-1_0.html) and IANA’s registry of JWT standard claims (https://www.iana.org/assignments/jwt/jwt.xhtml).

To enhance interoperability, the OpenID specification includes a feature called discovery (“OpenID Connect Discovery 1.0” by N. Sakimura, J. Bradley, M. Jones, and E. Jay, Dec 2023, https://openid.net/specs/openid-connect-discovery-1_0.html). OIDC discovery is a standard data structure that describes the configuration of an OpenID provider. It includes details like what OIDC claims they support (claims_supported), what’s the UserInfo endpoint (userinfo_endpoint), the authorization and token endpoints (authorization_endpoint and token_endpoint), the signing keys required to validate tokens (jwks_uri), and more. All OpenID providers expose this configuration under a /.well-known/openid-configuration endpoint. For example, Google Accounts is available under https://accounts.google.com/.well-known/openid-configuration.

7.7 Understanding role-based access controls

Role-based access controls (RBAC) are authorization checks based on user roles. Roles represent sets of permissions that define the level of access a user has to an application. Often, user roles correspond to the types of users an application has. For example, a healthcare application may distinguish between three types of users, such as patients, receptionists, and doctors, and use the concept of role to represent the different permission sets of each user type.

As explained in section 4.5, each user role may have access to specific operations or a role-dedicated API. For example, as illustrated in figure 7.15, a healthcare application may have a patient, doctor, and receptionist APIs. The challenge is how to ensure that each user has access to the right operations and APIs based on their role. How can users claims their association with a specific role, and how can APIs verify such association reliably?

Figure 7.15 A common solution for role-based access controls is to create distinct APIs for each user group or role, such as a doctor’s API, a patient’s API, and a receptionist’s API in this example.

A common solution to this problem is to include user roles in the token’s payload as a custom claim. Most identity providers support this feature, for example, Auth0 through the permissions custom claim and Microsoft Entra through the roles and groups custom claims. As illustrated in figure 7.16, when a user logs in with the identity server, the server knows their roles and includes them in the access token’s payload. In that case, the API server must validate that the user has the expected role when validating access to a certain API or operation. It is very important to ensure that each API and operation checks the user’s role before processing their requests, as failing to do so opens our API to broken function-level authorization breaches.

Figure 7.16 A common implementation for role-based access controls (RBAC) is to include the user role in the JWT through a custom claim, such as permissions in this example. The identity provider knows the user role and includes it in the token automatically, and the API validates user access by checking the permissions claim.

In addition to user roles, some APIs use the concept of granular access scopes. As illustrated in figure 7.17, we use granular access scopes to indicate which specific operations a user has access to. Granular access scopes help us simplify our RBAC model when every user has a different set of permissions, hence making it very difficult to create discrete user groups. Access scopes are often also represented in the token’s payload with a custom claim, making the authorization process explicit and transparent.

Figure 7.17 A popular implementation of RBAC is to define granular access controls. In this case, the token includes the list of endpoints the user has access to, and the API validates access by checking if the requested endpoint is in the permissions list.

For example, instead of having a single admin role, we may want to manage the permissions of our admin users dynamically and give each user access to different operations. Granular access scopes make it very explicit which operations a user has access to, hence making access validation easier on the server side. For example, as shown in figure 7.17, if a user is trying to perform an operation on the PUT /admin/api/users endpoint, we must validate if access to such endpoint is listed in their permissions list.

There are three main downsides to managing RBAC with granular access scopes: 1) the burden of maintaining each user’s list of permissions up to date, 2) the potential disclosure of sensitive configuration details in our security architecture, and 3) the potential for user permissions lists to grow very long. Point (1) can result in data breaches if a user loses access to one endpoint and we fail to update their access scopes list, while point (2) can reveal information that threat actors may be able to leverage to abuse our system. Point (3) can affect the size of our access tokens and have performance implications, or even exceed the maximum size of an HTTP request header allowed in some servers. Large lists of access scopes are a code smell and indicate that we can probably consolidate access scopes into discrete user groups or roles. As with everything in software design, balance the need for flexibility with the opportunity to simplify your processes by creating discrete groups of aggregated scopes.

Finally, some applications keep a table of user attributes and determine whether a user can access certain data or operations based on those attributes. We call this attribute-based access controls (ABAC). For example, as illustrated in figure 7.18, in a project management application, users may have different types of access depending on their job title, geographical location, and project affiliation. In this case, we pull the user ID from the access token and check their attributes against database to determine their level of access.

Figure 7.18 When we apply attribute-based access controls (ABAC), the API determines whether a user has access to the requested resource by checking their combination of attributes in the database.

ABAC gives us a lot of flexibility. Contrary to RBAC, we don’t represent ABAC in the access token, which means we don’t have the burden of maintaining user permissions in the authorization server up to date. However, this flexibility is also ABAC’s weak point. ABACs can get fairly complex very quickly, and they often build on top of implicit assumptions, which makes their implementation more difficult and testing and validation very challenging.

For example, consider the project management application in figure 7.18. Users can access data about projects at different levels of granularity. The application distinguishes between global, national, regional, state, county, and local project managers. For instance, a project manager with state-level access to California has access to all projects in that state. The application further distinguishes between read and write access depending on the business function. As you can see, access control rules quickly become complex, and often must account for special cases too, making the authorization process even more complicated. The downside of this complexity is that it becomes very difficult to test and maintain, and it can easily introduce side effects that threat actors can exploit to obtain unauthorized access. If you choose to use ABACs, make sure you create very explicit and detailed access models without any assumptions in them.

This concludes our overview of the most commonly used standards in API authentication and authorization. The topics we have laid out in this chapter are of utmost importance in API security, and sadly not always well understood. If you’re in the process of building a security framework for your APIs, make sure you’ve learned and understood all the material in this chapter to make informed decisions about your security model. If you already have a security model in place, evaluate your authentication and authorization processes with the help of this chapter, and see if you can spot any areas with room for improvement. Remember, API security is a process and there is always something we can do to improve our posture.

7.8 Summary

Authentication and authorization are the pillars of API security. They are our first line of defense, and all identity-based security checks rely on them, hence it’s important to get them right.

Authentication is the process of verifying a user’s identity, and authorization is the process of validating their access to an API, endpoint, or resource.

We prove that we have access to an API by including an access token in the request. An access token is a special string that goes in the Authorization request header.

The most common type of access token is the JSON Web Token standard (JWT)

JWTs contain a header, a payload, and a signature.

The header contains token’s metadata, such as the signing algorithm.

The payload contains information about the user. We call this information claims.

The signature proves that the token is legitimate.

Open Authorization (OAuth) is a standard for delegating access to user data to an application. For example, we use OAuth to allow a mobile application to access an API on our behalf.

OAuth describes five flows that can be used for delegating access depending on the application’s security constraints:

The authorization code flow is based on the use of a client secret, and hence it’s suitable for confidential clients such as web servers.

The proof of key code exchange (PKCE) flow is designed for public-source applications such as single-page applications. PKCE requires the client to prove ownership of the authorization code by providing a valid code verifier.

The client credentials flow uses credentials to obtain the token directly, and hence it’s suitable for scenarios that require programmatic access, such as microservices.

The device authorization grant is designed for input-constrained devices such as smart TVs. It requires the user to complete the flow through an external application.

To prevent JWT hijacking, we use sender-constrained tokens. We can bind the token to a user using two methods:

Mutual TLS for certificate-bound tokens binds the token to the user’s browser certificate.

Demonstrating proof of possession (DPoP) requires the client application to include a newly signed DPoP token on each request.

OpenID Connect (OIDC)is a standard that allows users to bring their identity from one application to another. OIDC allows us to outsource authentication to external identity providers to enhance the security of our APIs.

Role-based access controls (RBAC) allow us to define sets of permissions for groups of users, making it easier to validate their access to our APIs.

Attribute-based access controls (ABAC) allow us to constrain API access based on user attributes, such as the state of the user.

8 Implementing API authentication and authorization

This chapter covers

Documenting API security with OpenAPI

Issuing and validating JSON Web Tokens

Integrating with an OpenID Connect provider to add login and authorization to our APIs

Validating access tokens issued by an OpenID provider

Creating a middleware to authorize access to our APIs

Implementing role-based access controls

In February 2023, ethical hacker Eaton Zveare discovered major authentication and authorization vulnerabilities in Toyota’s Global Supplier Preparation Information Management System (GSPIMS), an application used by Toyota employees to manage their supply chain (https://eaton-works.com/2023/02/06/toyota-gspims-hack/). Among other things, Zvere was able to obtain access tokens without having to provide a password, impersonate Toyota employees, search the employee directory, assume administrator roles, and access highly sensitive data about Toyota’s supply chain. Thankfully, Zveare is an ethical hacker and he reported the vulnerabilities instead of exploiting them. However, in the hands of threat actors, vulnerabilities like this are the perfect recipe for a major data breach and put our whole business at stake.

Authentication and authorization are the cornerstones of our API security posture. As the Toyota example reveals, if threat actors break through our authentication and authorization system, they will be able to forge tokens, bypass access controls, impersonate other users, and get their hands on sensitive data. At that point, there is nothing much else we can do to stop threat actors; we’ve been breached. It’s therefore of utmost importance to implement a robust authentication and authorization system as the first line of defense for our APIs.

In chapter 7, you learned about the most widely used standards for API authentication and authorization. In this chapter, we implement those standards following all the recommendations and best practices from chapter 7. As we established in section 2.1, good security posture management begins with design and planning, and then documenting what we want to accomplish so that our documentation becomes an artifact that we can use to guide and validate our implementation. Following that principle, we’ll begin this chapter by learning to document authenticated operations in our API.

When we design our API, it’s important to decide which operations we want to protect and how, and document those decisions. We’ll tackle that task in the first section of this chapter. After that, you’ll learn to issue and validate JSON Web Tokens, then you’ll learn to integrate with an OpenID Connect (OIDC) provider to add login and authorization to your APIs, and you’ll learn to validate correctly the access tokens issued by the OIDC provider. Finally, you’ll learn to implement robust role-based access controls to restrict access to sensitive data and operations to authorized groups of users.

8.1 Documenting authenticated endpoints with OpenAPI

The first step in our API authentication and authorization strategy and deciding which operations we want to protect and how. For example, imagine a ride-sharing application. Do we want users to plan a ride and get a quote anonymously or should they be authenticated? Do we want to distinguish between different types of users, such as drivers and passengers, with access to different operations? Our API design process must answer those questions attending to the business requirements of our application, assessing the trade-offs of different choices, their security implications and how to manage them.

Once we have decided what to protect and how, we document our decisions. In this section, you learn to document authentication and authorization using OpenAPI, the most widely used standard for documenting REST APIs.

OpenAPI uses the concept of security schemes to describe how APIs are protected. A security scheme defines the process users must follow to authenticate their requests, including the authentication URL, the token type and its placement, access scopes, and more. To understand how we document security in OpenAPI, let’s break down the structure of an OpenAPI specification. The examples in this section use OpenAPI version 3, which is the latest version of OpenAPI.

Note

The semantics for security definitions are similar across all versions of OpenAPI, though slight variations do exist. For example, since version 3.1, OpenAPI is fully compatible with JSON Schema semantics, which allows you to describe richer and more complex data models. The examples in this section use OpenAPI 3.1.1. If you work with OpenAPI 2 and want to learn how to add security using that version, you’ll find a helpful guide on Swagger’s website (https://swagger.io/docs/specification/v2_0/authentication/authentication/). My recommendation, if you have OpenAPI 2 specifications, is to convert them to OpenAPI 3, which you can accomplish using Swagger’s OpenAPI Converter (https://converter.swagger.io/). OpenAPI 2 is now deprecated, and you’ll find more community and better tooling support if you migrate to OpenAPI 3.

As illustrated in figure 8.1, an OpenAPI specification contains the following sections:

openapi: the version of OpenAPI the specification uses. For example, 3.1.1.

info: metadata about the API described in the specification, like the title and version.

servers: the URLs where the API is available, like the production and development environments.

paths: the URL paths exposed by the API, together with the HTTP methods they support, the status codes they return, the content types they use, etc.

components: a collection of reusable elements that can be referenced throughout the specification. For example, request and response schemas are often documented within this section, under components.schemas. Common API responses, like 404 and 422 responses, are also documented in this section, under components.responses. Security schemes also fit within this section under components.securitySchemes. To learn more about the elements that go into the components section, check out the useful guide to 3.0 by Swagger (https://swagger.io/docs/specification/v3_0/components/).

security: the security schemes that by default apply to the whole API.

Figure 8.1 We define the security schemes that protect our API in the specification’s components section, under securitySchemes, and we apply globally to the API under the specification’s security field, or per operation, using the operation’s security field.

As illustrated in figure 8.1, we define security schemes within the components section of the OpenAPI specification, right under securitySchemes. A security scheme object describes how we must authenticate the request, including the type of token and where we must place it. The exact shape of the security scheme object depends on the type of security. OpenAPI 3 supports five types of security schemes:

http: authentication via the Authorization header, as per RFC 7235 (“Hypertext Transfer Protocol (HTTP/1.1): Authentication” by R. Fielding and J. Reschke, June 2014, https://datatracker.ietf.org/doc/html/rfc7235). OpenAPI distinguishes two schemes of HTTP authentication: Basic and Bearer.

apiKey: authentication with API keys.

oauth2: describes how to obtain an access token, the supported OAuth flows, and the available scopes.

openIdConnect: describes the OpenID discovery endpoint.

mutualTLS: authentication over mutual TLS, which requires clients to use the TLS certificate bound to their identity.

Let’s see an example of documenting a security scheme for the http security type. Listing 8.1 defines a security scheme named JWTBearer, which describes how to authenticate requests with JSON Web Tokens as bearer tokens. A bearer token is a token with the Bearer keyword before the token, with the following format: Authorization: Bearer <token>. The security scheme definition in listing 8.1 includes the following fields:

description: a short description of the security scheme.

type: identifies the security scheme as an HTTP authentication type.

scheme: the security scheme to include in the Authorization header. It must be one of the values listed under IANA’s “HTP Authentication Scheme Registry” (https://www.iana.org/assignments/http-authschemes/http-authschemes.xhtml).

bearerFormat: the bearer token’s format, which in listing 8.1 is JWT.

Listing 8.1 OpenAPI JWT Bearer security scheme

# file: ch08/openapi.yaml
components:
  securitySchemes:
    JWTBearer:    #A
      description: Bearer JSON Web Token
      type: http
      scheme: Bearer
      bearerFormat: JWT

To learn more about documenting other security schemes, check out OpenAPI’s official documentation, in particular “Describing API security” (https://learn.openapis.org/specification/security.html) and section 4.8.27 of the OpenAPI 3.1.1 specification, which describes the requirements for security scheme objects (https://spec.openapis.org/oas/latest.html#security-scheme-object-0).

Once we’ve defined our security schemes, we apply them to the API. We can apply a security scheme globally, using OpenAPI’s top-level security keyword, as shown in figure 8.1, or we can apply them per operation, using the security keyword available on each operation. Operation-level security overrides the global security scheme. Let’s see an example of each.

Listing 8.2 defines an API that applies the JWTBearer security scheme globally. The JWTBearer security scheme is the same as in listing 8.1, hence it’s collapsed in listing 8.2. The API exposes two endpoints: GET /hello and GET /protected-hello. GET /protected-hello is protected by the JWTBearer security scheme since it inherits its security configuration from the specification’s top-level security property. GET /hello is unprotected since it overrides the specification’s global security configuration with its own, setting it to an empty array, which effectively disables security for that particular endpoint.

Listing 8.2 Applying a security scheme globally

# file: ch08/openapi.yaml
openapi: 3.1.1
security:
  - JWTBearer: []    #A
paths:
  /hello:
    get:
      security: []    #B
      ...
  /protected-hello:
    get:
      ...
components:
  securitySchemes:
    JWTBearer:
      ...

Documenting the security schemes and configurations of our API is a fundamental step to allow us to manage our API security posture, model threats to our API, and understand our attack vectors. Once we’ve documented our security, it’s time to implement it, and that’s the topic of the rest of this chapter!

8.2 Issuing JSON Web Tokens

JSON Web Tokens (JWTs) are JSON objects that contain claims about the right of a user to access our API. JWTs are a well-established standard and the most common type of token used to validate access to APIs. However, using JWTs alone won’t make your API more secure: the key is to have a robust process for issuing and validating JWTs securely and following best practices. That’s what this section is about.

Let’s talk about issuing JWTs first. You can issue JWTs yourself, or you can delegate the job to an identity provider (IdP). As we discussed in chapter 7, identity systems are difficult to build and secure, and most organizations stand to benefit from using a standard IdP. If you use an IdP, the provider will take care of issuing JWTs securely, and you only need to worry about validating JWTs correctly.

If using an IdP is not an option and you do need to issue your own tokens, use self-hosted IdP such as Keycloak (https://www.keycloak.org/), and if that’s not possible, use a robust library for issuing tokens. Most software development stacks have a good ecosystem of libraries that implement the JWT standard, and jwt.io maintains a catalog of them (https://jwt.io/libraries), including information such as the signing algorithms they support, the version of the library we can use, and links to libraries’ code repositories and documentation.

For Python, we have three outstanding libraries: PyJWT (https://github.com/jpadilla/pyjwt), python-jose (https://github.com/mpdavis/python-jose/, no connection with truly yours), and JWCrypto (https://github.com/latchset/jwcrypto). PyJWT is the most popular library. It has an easy-to-use interface and supports a wide range of signing algorithms. We’ll use this library to illustrate the examples in this chapter, but you can find examples for the other libraries in the code repository for this chapter too.

To generate a JWT, we need a payload. As we learned in section 7.2, JWTs contain three elements: a header, a payload, and a signature. The header contains metadata about the token and will be generated automatically by PyJWT. The payload contains the claims about the user, and we must provide those claims to PyJWT. We’ll also need a signing key, which PyJWT will use to sign the token.

Let’s begin by putting together our payload, as shown in listing 8.3. The JWT payload in listing 8.3 contains a few standard claims: it identifies the token’s issuer (iss) as https://auth.apithreats.com, provides a user identifier under the sub field, states the audience (aud) for which this token is intended (https://apithreats.com/api), it sets the token’s issue time (iat) to the current time in UTC, and set its expiration (exp) to an hour. The payload can include additional number of custom properties as needed, for example to include information such as user permissions, roles, and other relevant information. Make sure you don’t include personally identifiable information (PII), such as the user’s personal name, email address, and such, since that poses a risk of PII leak as we discussed in section 7.2.1.

Listing 8.3 Example of JWT payload

# file: ch08/jwt_generator_pyjwt_hs256.py
from datetime import datetime, timedelta

now = datetime.utcnow()
payload = {
    "iss": "https://auth.apithreats.com",
    "sub": "23456543",
    "aud": "https://apithreats.com/api",
    "iat": now,
    "exp": (now + timedelta(hours=1)).timestamp()
}

Now that we have a payload, let’s feed it to PyJWT to issue a token. Listing 8.4 shows the process. We begin by importing datetime() and timedelta() from the datetime library, which will help us set the token’s issued and expiration times. We also import the PyJWT library, which we’ll use to produce the token. In lines 6-13, we define the payload that we are going to feed to PyJWT’s encode() function to issue the token. Then we invoke PyJWT’s encode(), passing the token’s payload, the desired signing algorithm (algorithm), and the signing key (key). In this example, we sign the token with HS256 using the secret password as our hashing key. The secret is hardcoded in the script for illustration purposes – in the real world, you’d pull it from the environment.

Listing 8.4 Issuing a JWT with PyJWT

# file: ch08/jwt_generator_pyjwt_hs256.py
from datetime import datetime, timedelta
import jwt

now = datetime.utcnow()
payload = {
    "iss": "https://auth.apithreats.com",
    "sub": "23456543",
    "aud": "https://apithreats.com/api",
    "iat": now,
    "exp": (now + timedelta(hours=1)).timestamp()
}

print(jwt.encode(payload=payload, key="password", algorithm="HS256"))

Listing 8.4 is available in the code repository for this book under ch08/jwt_generator_pyjwt_hs256.py. To run the script, open a terminal and follow the instructions in the chapter’s README (ch08/README.md) to install the necessary dependencies and activate the virtual environment. When you execute the command, you’ll see the token issued by PyJWT printed on the terminal:

$ python jwt_generator_pyjwt_hs256.py
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2F1dGguYXBpdGhyZW
[CA]F0cy5jb20iLCJzdWIiOiIyMzQ1NjU0MyIsImF1ZCI6Imh0dHBzOi8vYXBpdGhyZWF0cy5jb
[CA]20vYXBpIiwiaWF0IjoxNzMxNTY4Nzg0LCJleHAiOjE3MzE1NzIzODQuODMxMDE3fQ.i1d4M
[CA]eudVHhpNG8fyfLamK8v01hhU_BHe7-OO_qDISo

You can paste the token on https://jwt.io to inspect its contents, as shown in figure 8.2. jwt.io shows you the decoded content of the token’s header and payload, and allows you to verify the signature by typing the key within the “Verify signature” box.

Figure 8.2 jwt.io is an online tool that makes it easy to work with tokens. When you paste the token on the left-side panel, the decoded header and payload appear on the right-side panel. You can also verify the token’s signature by pasting the secret within the “Verify signature” field.

Listing 8.4 uses HS256 to sign the token, which, as we learned in section 7.2.1, is a type of symmetric signing algorithm. Symmetric algorithms use the same key to sign the token and validate its signature, which poses a risk of leaking the signing key since we need to have it in multiple places. To mitigate that risk, we use an asymmetric signing algorithm. The most widely supported asymmetric signing algorithm for JWT implementations is RS256. As we saw in section 7.2.1, RS256 is vulnerable to Bleichenbacher’s ’06 signature forgery attack when

the signing key has a low exponent (below 3) and

there are flaws in the implementation of the verification algorithm.

Barring those problems, RS256 is safe to use. However, implementation flaws in the validation process are concerning enough that the FAPI specification (https://openid.net/wg/fapi/specifications/), which is a set of guidelines for building highly secure APIs, bans the use RS256 and recommends using more robust alternatives like PS256 and EdDSA (“Financial-grade API Security Profile 1.0 - Part 2: Advanced” by N. Sakimura, J. Bradley, and E. Jay, March 2021, https://openid.net/specs/openid-financial-api-part-2-1_0.html, 8.6; see also chapter 10 for an introduction to the FAPI specification). PS256 is like RS256 but with probabilistic outcomes, i.e. for the same input, we get different signatures. This substantially mitigates the risk of an attacker being able to forge tokens. Here, we’ll see an example for generating tokens signed with PS256. The book’s repository contains examples for EdDSA and RS256 too if you want to learn how to generate tokens with such signatures.

Asymmetric signing algorithms use a private key for signing and a public key for signature validation. Before we can sign tokens with asymmetric keys, we have to produce our public and private keys. Since we are going to illustrate the signature process with PS256, which uses RSA keys, we need to generate an RSA key pair, for which we can use OpenSSL (https://github.com/openssl/openssl). OpenSSL is a library that implements the SSL and TLS protocols, and among other things, it allows us to generate RSA keys.

To install OpenSSL, please refer to the “Build and Install” section of the README in OpenSSL’s official GitHub repository (https://github.com/openssl/openssl/tree/master?tab=readme-ov-file#build-and-install), and find instructions for your Operating System. If you’re unable to install OpenSSL or want to do it later, the GitHub repository for this book includes a sample of private and public keys under ch08/private_key.pem and ch08/public_key.pem that you can use for your own testing purposes.

Once you have installed OpenSSL, you can generate your RSA private and public keys using OpenSSL’s genpkey command, which is the recommended way to generate signing keys (https://docs.openssl.org/3.2/man1/openssl-genpkey/). It looks like this:

openssl genpkey -algorithm RSA -out private_key.pem
[CA] -outpubkey public_key.pem -pkeyopt rsa_keygen_bits:3072

Let’s break down the command to understand what’s going on. We have the following parameters:

-algorithm indicates the algorithm we want to generate the key for, which in this case is RSA.

-out specifies the file where we want to output the private key, which in this case is private_key.pem. This is the key that we’ll use to sign tokens.

-pubout specifies the file where we want to output the public key. This is the key we’ll use to validate signatures.

-pkeyopt allows us to provide additional configuration for our key. In this case, we configure the size of the key with the rsa_keygen_bits parameter and we set it to 3072 bits. The strength of an RSA key is proportional to its size, and RFC 7518 (“JSON Web Algorithms (JWA)” by M. Jones, May 2015, https://www.rfc-editor.org/rfc/rfc7518.html) requires a size of at least 2048 bits.

The previous command produces two files: private_key.pem and public_key.pem, which are our private and public keys respectively. Now that we have a key pair to sign and validate token signatures, let’s see how we feed them to PyJWT. The first step is to load the private key using Python’s cryptography library, as shown in listing 8.4. We begin by reading the private_key.pem file’s contents using the Path class from Python’s pathlib library. pathlib contains handy utilities to work with files, including reading from and writing to them. In the next step, we use the load_pem_private_key() function from cryptography’s serialization module to load our private key as an object. load_pem_private_key() takes two required parameters: data and password. data is the content from our private key file in bytes. In the previous step, we loaded the file content and assigned it to a variable named private_key_text, and to encode it, we use the encode() method available to all Python strings. password is only required if our key is encrypted, which in this case is not, so we set it to None.

Listing 8.5 Loading a private key with Python’s cryptography library

# file: ch08/jwt_generator_pyjwt_ps256.py
from pathlib import Path
from cryptography.hazmat.primitives import serialization

private_key_text = Path(    #A
    "private_key.pem"
).read_text()

private_key = serialization.load_pem_private_key(    #B
    data=private_key_text.encode(), password=None
)

Now that our key is loaded as an object, we can pass it to PyJWT to sign our tokens. Listing 8.6 shows how we do it. The listing uses the same payload we defined in listing 8.4, so here it appears collapsed with an ellipsis. To produce the token, we use PyJWT’s encode() function, passing the token’s payload to the payload parameter, the private key object to the key parameter, and selecting the algorithm we want to use to sign the token through the algorithm parameter. In this case, we sign the token using PS256. If you wish to sign with RS256, you can use the exact same code from listing 8.6, replacing PS256 with RS256 in the last line of the listing.

Listing 8.6 Loading a private key with Python’s cryptography library

# file: ch08/jwt_generator_pyjwt_ps256.py
from pathlib import Path
import jwt
from cryptography.hazmat.primitives import serialization

private_key_text = Path("private_key.pem").read_text()

private_key = serialization.load_pem_private_key(
    data=private_key_text.encode(), password=None
)

payload = {...}  # same as in listing 8.3

print(
    jwt.encode(payload=payload, key=private_key, algorithm="PS256")
)    #A

The code in listing 8.6 is available under ch08/jwt_generator_pyjwt_pss256.py in the code repository for this book. If you run the script, it prints the token to the terminal:

$ python jwt_generator_pyjwt_ps256.py
eyJhbGciOiJQUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2F1dGguYXBpdGhyZW
[CA]F0cy5jb20iLCJzdWIiOiIyMzQ1NjU0MyIsImF1ZCI6Imh0dHBzOi8vYXBpdGhyZWF0cy5jb
[CA]20vYXBpIiwiaWF0IjoxNzMxNjU2NzI1LCJleHAiOjE3MzE2NjAzMjUuNjA1NzQ2fQ.ViiRA
[CA]7U-0RwkAgAqukBSugkS6rT8uR-AdPhp95YiaNs-wQSPg8H3HPGqExJDIwtWqeZIiMLtYEdl
[CA]VyDfYHQ4xJYY9LsGqWkXUbKlauBVoaey1SfrAbXK7N8zTzchZJciNjehvxUCjYr10ePzKbd
[CA]Gwf05hYle03fNQ_uFjF4Ooosr9100lAM-JBwWt95gzNQnS-DMfdYTdS74Y4Ri6wVfsyquzh
[CA]h4KT_Y86OelSWvRChY_IL16t38XMM_UfnW1T0ApolKSA8aeWQlhUNb0QBPn2CPVg6TR4ca1
[CA]IYJPSNNfbn0zkcJPHevh2quNuIji4dFp1up_BdATqIGKV9-LTRJeAHfhsHobhPgw5AW60Nm
[CA]cVHmaUtCc2YMCWWphSKtOA6RAXNyU9Ko8Vx_0GNpoxP4Q1v55YZq5aAIV7k9u0m5oYRnabb
[CA]PWplX1DAWCgpvVM2GZScdpOfn-4eqfYH8YZ5ULg3pn5qoAYV_dZLyZVpTxefvnaCIxU-9D4
[CA]1DU84YVxWR

You can paste the token on jwt.io to verify that the content is right, as we did earlier in figure 8.2.

Now that you can sign tokens using RS256 and PS256, you can check what we mean when we say that RS256 is deterministic while PS256 is probabilistic. First, create a hardcoded payload (i.e. without the dynamic timestamp values) like the following:

payload = {
    "iss": "https://auth.pyjobs.works",
    "sub": "10",
    "aud": "https://pyjobs.works/jobs",
    "iat": 1825977600,
    "exp": 1826064000
}

Replace this payload in the ch08/jwt_generator_pyjwt_ps256.py script and run it multiple times. You’ll see that it produces a token with a different signature each time. However, if you run the script using the RS256 algorithm, you’ll see that it always produces the exact same signature.

Now that we know how to generate JSON Web Tokens, let’s see how we validate them!

8.3 Validating JSON Web Tokens

When we use JSON Web Tokens to authenticate API requests, a big part of our security posture relies on our token validation process. In other words, JWTs are only as secure as how we validate them. Most security breaches involving JWTs happen in the context of token validation. For example, in December 2020, cybersecurity researcher Ron Chan revealed how Microsoft Outlook failed to validate token signatures, allowing him to forge tokens and access other users’ emails (https://youtu.be/t54N4x2uIPs). In April 2020, Ben Knight, a cybersecurity researcher at Insomnia Security (now part of CyberCX), revealed how Auth0 became vulnerable to token forgery by setting the token’s alg field to nonE, with E at the end (https://cybercx.co.nz/blog/json-web-token-validation-bypass-in-auth0-authentication-api/). These are two examples of a long list of security issues involving JWT validation.

Given the track record of API breaches involving JWTs and the types of companies involved, including the likes of Microsoft and Auth0, one would think that validating JWTs is inherently difficult. In fact, this is not the case. As you’ll see in this section and the rest of this chapter, the key to validating JWTs securely is to keep the process simple and apply a few good security practices.

Note

When choosing a JWT library, make sure it follows best practices and avoids common mistakes in token validation. For example, in 2015, cybersecurity researcher Tim McLean found critical vulnerabilities in a few open-source JWT implementations that allowed threat actors to set the alg field of the token’s header to none and therefore bypass signature validation. He also found libraries vulnerable to algorithm confusion attacks, which allowed attackers to bypass asymmetric signature validation by changing the token’s alg field to HS256, which made the libraries think the public key was the signing key (https://www.chosenplaintext.ca/2015/03/31/jwt-algorithm-confusion.html). I’m not aware of any major JWT libraries currently vulnerable to these attacks, but it’s always a good idea to test your library of choice using McLean’s examples to verify it’s not vulnerable.

Let’s begin with a simple example of validating a token signed with HS256 to understand how the process works. Run the script from listing 8.4, also available in the code repository for this book under ch08/jwt_generator_pyjwt_hs256.py. Copy the token from the terminal and replace it with the <token> placeholder in listing 8.7. The listing defines a function validate_token() that takes one parameter – the access token. Within the function, we invoke PyJWT’s decode() function and return its outcome. PyJWT’s decode()validates and decodes the token. If the validation is successful, we get the token’s payload back, and if it fails, it raises an exception.

PyJWT’s decode()takes three required parameters:

jwt: the JSON Web Token we want to validate.

key: the key we want to use to validate the token’s signature.

algorithms: the list of algorithms we can use to validate the token.

Additionally, we specify the allowed values for the token’s audience and issuer (aud and iss claims in the payload). PyJWT also validates that the token isn’t expired.

Listing 8.7 Validating tokens with HS256 signature

# file: jwt_validator_pyjwt_hs256.py
import jwt

def validate_token(token):
    return jwt.decode(    #A
        jwt=token,
        key="password",    #B
        algorithms=["HS256"],    #C
        audience="https://apithreats.com/api",
        issuer="https://auth.apithreats.com",
    )

token = <token>
print(validate_token(token))

The code in listing 8.7 is available in the code repository for this book under ch08/jwt_validator_pyjwt_hs256.py. If you run the script with a valid token, it will print the token’s payload to the terminal. To get a sense of the type of errors PyJWT raises when the token is invalid, try changing the audience’s value or the list of allowed algorithms in listing 8.7, and run the script again.

Validating tokens signed with asymmetric algorithms, like RS256 and PS256, follows the same process as in listing 8.7, but we need to load the public key first. Listing 8.8 shows how to do it. Listing 8.8 is very similar to listing 8.7, with the difference between the two highlighted in bold case. To load the public key, we read the contents from our public_key.pem file and pass them on encoded as bytes to cryptography’s load_pem_public_key() function.

Listing 8.8 Validating tokens with PS256 signature

# file: jwt_validator_pyjwt_ps256.py
from pathlib import Path
import jwt
from cryptography.hazmat.primitives.serialization \
    import load_pem_public_key

public_key = load_pem_public_key(    #A
    Path("public_key.pem").read_text().encode()
)

def validate_token(token):
    return jwt.decode(
        jwt=token,
        key=public_key,    #B
        algorithms=["PS256"],    #C
        audience="https://apithreats.com/api",
        issuer="https://auth.apithreats.com",
    )

token = <token>
print(validate_token(token))

The code from listing 8.8 is available in the code repository for this book under ch08/jwt_validator_pyjwt_ps256.py. To run the script, generate a PS256 signed token first using the code from listing 8.6, and paste the token into listing 8.8 replacing the <token> placeholder. If the token is valid, you’ll get the token’s payload back, and otherwise PyJWT will raise an appropriate error.

That’s all it takes to validate tokens correctly! Let’s sum up the best practices we’ve seen in this section to validate tokens securely and avoid common mistakes:

Use a good JWT library with support for all the algorithms you need. In particular, make sure the library supports strong signing algorithms like PS256 and EdDSA. Don’t compromise on this point – if a library doesn’t support the algorithms you need, choose another one. Also, make sure the library has a large community. A large user-based means there are many eyes on the library, it’s battled-tested, and any issues will likely be reported as soon as they come up.

Make sure the JWT library you use isn’t vulnerable to algorithm confusion attacks. Run your own tests following Tim McLean’s examples against your JWT library of choice to verify it’s free of such vulnerabilities.

If you use RS256, make sure the library isn’t vulnerable to Bleichenbarcher’s ’06 attack. Also, make sure your RSA signing keys are secure with a high exponent and a minimum size of 2048 bits.

Hardcode the allowed algorithm in your token validation function, i.e. the allowed value in the token header’s alg field. This avoids none attacks against your tokens and variants of it, like the nonE vulnerability that was found earlier in Auth0.

Validate all the claims you can, including the token’s issuer, audience, and so on. If your JWT library doesn’t allow you to automatically pass additional claims for validation, as in the case of PyJWT, add your own assertion to your JWT validation function.

If you follow these practices and stick to the validation process outlined in this section, your JWT-based authentication will be as robust and secure as it can be. Don’t overcomplicate it, keep it simple, and stick to the principles.

So far, we’ve learned to issue and validate JWTs using our own self-generated signing keys. The process is straightforward: we use the private key to sign the token, and the public key to validate the signature and all the claims in the token’s payload. But what about situations when we don’t issue the JWTs ourselves? For example, when we work with an external Identity Provider (IdP)? How does the process work in that case? If this is your case, the next section will answer all your questions.

8.4 Integrating with an OpenID Connect provider

A common solution to delivering secure and reliable identity management in our APIs is to outsource the job to a third-party Identity Provider (IdP) like Auth0, AWS Cognito, Curity, Microsoft Entra, and similar services. These services take care of managing the identity of our users securely, issuing robust access tokens in the form of JWTs and other formats, securely managing and rotating our signing keys, and more. If you can’t outsource identity to an IdP, consider using a self-hosted IdP like Keycloak (https://www.keycloak.org/) instead of building your own solution. In this section, we go through the steps required to set up our system to issue and validate securely JWTs from an external IdP.

We’ll focus the discussion specifically on IdPs that implement OpenID Connect (OIDC), as that’s the most widely used standard for user authentication. We’ll use a specific example of integration with Auth0; the example is generalizable since all OIDC providers work in very similar ways. Check out Appendix B for a step-by-step guide to setting up Auth0 if you want to work through all the examples in this chapter. You’ll need an Auth0 setup to be able to issue your own tokens.

The code repository for this book includes a simple API under ch08/api.py. ch08/api.py is built with Python’s popular FastAPI framework and runs on port 8000. We’ll add authentication and the ability to issue access tokens to that API using our integration with Auth0. Specifically, we’ll implement the authorization code flow, and Appendix B guides you through the process of configuring an authorization code client on Auth0. We’ll also add the logic necessary to validate access tokens. Please check out ch08/README.md for details on how to run the run the API. Throughout this section, we’ll illustrate some concepts using apithreats.com’s discovery endpoint, since it’s also implemented with Auth0 and it’s publicly available.

As we learned in section 7.6, systems that implement OIDC publish the information that we need to integrate with them in what we call the discovery endpoint, which is a URL with path /.well-known/openid-configuration. For example, Google Identity’s discovery endpoint is https://accounts.google.com/.well-known/openid-configuration. The discovery endpoint tells us how we initiate an authorization request, how we obtain access tokens, how we validate them, and more. If you use Auth0 as you IdP, your discovery endpoint is usually your tenant’s URI followed by the discovery endpoint’s path (/.well-known/openid-configuration). For example, apithreats.com authentication is implemented with Auth0, and its discovery endpoint is https://apithreats.eu.auth0.com/.well-known/openid-configuration.

8.4.1 Log in users and issue access tokens with an OpenID Connect provider

We’ll first add login and the ability to issue access tokens to our API. Auth0 manages the user login process, so we need to redirect our users to Auth0 to verify their identity and obtain an access token. This is what we call an authorization request. In this section, we implement the authorization request with the authorization code flow using the Auth0 client configured in Appendix B. The process that we are going to implement to issue access tokens is illustrated in figure 8.4. Take your time to study figure 8.4, since it represents the whole flow of requests involved in issuing a token. Let’s briefly analyze every step in the flow so that we understand what’s going on:

Step 1: the user visits our server's GET /login page.

Step 2: our server redirects the user to Auth0’s login page using the authorization request URL.

Step 3: the user visits Auth0’s login page using the authorization request URL.

Step 4: Auth0 responds with the login page.

Step 5: the user logs in by sending their credentials with a POST request.

Step 6: upon successful login, Auth0 asks the user if they want to grant the client application (in this case the server) access to their data.

Step 7: the user grants consent for the client application to access their data.

Step 8: Auth0 redirects the user back to the API server including the authorization code in the URL.

Step 9: the user requests the server to exchange the authorization code for an access token.

Step 10: the server exchanges the access token plus the client_id for an access token.

Step 11: Auth0 issues the access token and the server sends it to the user.

Figure 8.3 Flow of requests involved in obtaining an access token. The process begins with the user visiting server’s login page. The API redirects the client to Auth0’s login screen, where the user authenticates and grants consent, and is then redirected back to the API with the authorization code.

As illustrated in figure 8.4, when the user visits the GET /login URL in our server, we must redirect them to Auth0’s login screen using the authorization request URL. Our first task then is to figure out how we construct the authorization request URL. In the discovery endpoint’s response, there’s a field called authorization_endpoint, which is the URL we must use to initiate the authorization request. For apithreats.com, the authorization endpoint is https://apithreats.eu.auth0.com/authorize. From the OAuth 2.1 specification (https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-12), section 4.1.1, we know that the authorization request must include the following parameters:

response_type: the type of response we’ll receive from the authorization request. The acceptable values are defined within the response_types_supported field in the discovery endpoint’s response. For example, to obtain an access token, the value is code, and if we want to obtain an access token and an ID token, the value is code id_token.

client_id: the ID of our client application.

If we have multiple APIs configured in our IdP, we’ll also need to specify the audience, i.e. the API we want to access, and if we have multiple redirect URIs configured, we must also specify which one we want to use. The full authorization request looks like this:

https://apithreats.eu.auth0.com/authorize?response_type=code
[CA]&audience= https://apithreats.com&client_id={client_id}
[CA]&redirect_uri=http://localhost:8000/docs

We parametrize the client_id so that we can run the application with different clients, like a test client, a development client, and a production client. In this case, we’re running the application with a test client that allows us to run the whole process locally, hence the redirect_uri points to the local host. With this information at hand, we can now add a login route to our API, as shown in listing 8.9.

We begin listing 8.9 by importing the utilities we’ll need from Python’s os module, FastAPI, and starlette. Since we’re parametrizing the client_id, we attempt to pull its value from an environment variable named AUTH0_CLIENT_ID, and run an assertion to check it’s been set, otherwise we exit the program immediately. The ellipsis omits the API implementation for which we are adding authentication, which you can check in file ch08/api.py in the code repository for this book. Finally, we create a GET /login endpoint that redirects the user to the authorization endpoint with all the expected parameters set.

Listing 8.9 Make an authorization request with the authorization code flow

# file: ch08/api.py
import os
from fastapi import FastAPI
from starlette.responses import RedirectResponse

client_id = os.getenv("AUTH0_CLIENT_ID")    #A
assert client_id is not None, "AUTH0_CLIENT_ID environment variable needed"

server = FastAPI()
...
@server.get("/login")    #B
def login():
    return RedirectResponse(    #C
        "<authorization_endpoint>"
        "?response_type=code"
        f"&client_id={client_id}"
        "&redirect_uri=http://localhost:8000/docs"
        "&audience=<audience>"
    )

To try out the code in listing 8.9, make sure to replace the <authorization_endpoint> placeholder with your own tenant’s authorization endpoint, and the <audience> placeholder with your own audience. Then, cd into the ch08/ directory and run the following command to start up the API server:

uvicorn api:server --reload

Once the server is up and running, open a browser and visit http://localhost:8000/login, which will take you to the login page. You’ll need to create an account first. After successfully logging in, you’ll be redirected back to http://localhost:8000/docs. If you check the URL, you’ll notice there’s a code parameter in the URL, as shown in figure 8.3. That’s the authorization code, and we’ll need to exchange it for an access token.

Figure 8.4 After successful login, Auth0 redirects the user back to our site. In the URL, you’ll see a code parameter, which contains the authorization code that we must exchange for the access token.

To exchange the authorization code for an access token, we need to send a request to the token endpoint. The token endpoint’s URL is available in the discovery endpoint’s response under the token_endpoint field. For apithreats.com, the token endpoint is https://apithreats.eu.auth0.com/oauth/token. The specific semantics of the request vary according to the provider and the authorization flow. For the authorization code flow, Auth0 requires the following parameters:

grant_type: the authorization flow. For the authorization code flow, the value is authorization_code.

client_id: the ID of our client application.

client_secret: our client application’s secret.

code: the authorization code.

redirect_uri: the same redirect URI we passed in the authorization request.

We send those parameters over a POST request using the x-www-form-urlencoded MIME type, as shown in listing 8.10. Listing 8.10 extends the code from listing 8.9 by adding a new endpoint that allows us to exchange the authorization code for an access token. The newly added code is highlighted in bold, and some of the previous code is omitted with ellipses. We pull the client ID from an environment variable named AUTH0_CLIENT_SECRET, and like we did earlier, we check with an assertion that it’s correctly set before proceeding. The newly added GET /token endpoint requires a URL query parameter named code, which is the authorization code, and sends it together with the other required parameters to the IdP’s token endpoint. Finally, we return the response from the authorization endpoint.

Listing 8.10 Exchange the authorization code for an access token

# file: ch08/api.py
import os

import requests
from fastapi import FastAPI
from starlette.responses import RedirectResponse

client_id = os.getenv("AUTH0_CLIENT_ID")
client_secret = os.getenv("AUTH0_CLIENT_SECRET")    #A

assert client_id is not None, "AUTH0_CLIENT_ID environment variable needed"
assert (
    client_secret is not None,
    "AUTH0_CLIENT_SECRET environment variable needed"
)

server = FastAPI()
...

@server.get("/token")    #B
def get_access_token(code: str):    #C
    payload = (    #D
        "grant_type=authorization_code"
        f"&client_id={client_id}"
        f"&client_secret={client_secret}"
        f"&code={code}"
        "&redirect_uri=http://localhost:8000/docs"
    )
    headers = {    #E
        "content-type": "application/x-www-form-urlencoded"
    }
    response = requests.post(    #F
        "<token_endpoint>",
        payload, headers=headers
    )
    return response.json()    #G

To test the new endpoint, replace the <token_endpoint> placeholder with your own tenant’s token endpoint. Then, start up the API server and log in again. You’ll be redirected back to http://localhost:8000/docs, with the authorization code displayed in the URL, as we saw earlier in figure 8.3. Copy the code, and send it to the newly added /token endpoint using the Swagger UI, as shown in figure 8.5, to obtain your access token.

Figure 8.5 To exchange the authorization code for an access token, copy the code’s value and paste it into the code field of the GET /token endpoint. Then, hit the “Execute” button, and you’ll get the API response with the access token in it.

Now we can issue access tokens for our API! The next step is validating those access tokens to protect our API from unauthorized requests. That’s the topic of the next section.

8.4.2 Validating access tokens issued by an OpenID Connect provider

Validating access tokens from an IdP follows the same process we saw in section 8.3 to validate tokens signed with our own signing keys: we’ll need to load the public key to validate the signature and validate all the claims in the token’s payload. Of course, in this case, we don’t have a public key laying around in our file system, so how do we find it? The discovery endpoint provided by our IdP contains a JWKS endpoint under the field jwks_uri. The JWKS endpoint contains the list of public keys available to validate signatures in the tokens issued by the IdP. For example, for apithreats.com, the JWKS endpoint is https://apithreats.eu.auth0.com/.well-known/jwks.json.

DEFINITION

JSON Web Key (JWK) is a standard JSON data structure for representing cryptographic keys. The fields in the JWK object contain properties of the cryptographic key that are needed to load and utilise the key correctly to validate a signature. For example, the kty field indicates the type of key, and the kid field indicates the ID of the key. OIDC providers publish the keys available for validating token signatures in the JWKS endpoint. To learn more about the JWK specification, check out RFC 7518, “JSON Web Algorithms (JWA)” by M. Jones, May 2015 (https://www.rfc-editor.org/rfc/rfc7518.html).

The JWKS endpoint’s response contains an array of keys. Here’s an example of JWK from apithreats.com:

{
  "kty": "RSA",
  "use": "sig",
  "n": "1F72TCRTs6TOXp3pN3sczKIw6OlZtuu_IV5nyrv3OpXEwuf_kOxmd-Gk2F1AwfTdUdH
  [CA]MoMa5whwYVTemamUPDcmm_mkSLdKYscGWjsWAJc6BewO2rtecqk_BCWwQ5EVFreAjQVFeq
  [CA]yUHvg8YJX40T_KFsHAhiOzVp5vxSPxdyKQFTErYS8e4Pwf5rigPVk02IkKPYOE9U71fYFi
  [CA]eP9OHVbTM34qTzR3dekc9CPz5P05ZFSSs32ZfOCXtMEhyJB2ky5LHqzjbAmE8ahXdZbo9V
  [CA]U9juk4THKbHa-s1_1El8frRwGm3FANMePZDGNT73h-Q6n_QFEhewv-dMWHdCB8-NQ",
  "e": "AQAB",
  "kid": "080E8CtRFAnnLlgK3dk8Y",
  "x5t": "bGbu8vMg1fEwLf8eJ-pEltvuEKw",
  "x5c": ["MIIDCTCCAfGgAwIBAgIJAqiJX3yiNIe0MA0GCSqGSIb3DQEBCwUAMCIxIDAeBgNVB
  [CA]AMTF2FwaXRocmVhdHMuZXUuYXV0aDAuY29tMB4XDTI0MDMxMzA5MjYxMVoXDTM3MTEyMDA
  [CA]5MjYxMVowIjEgMB4GA1UEAxMXYXBpdGhyZWF0cy5ldS5hdXRoMC5jb20wggEiMA0GCSqGS
  [CA]Ib3DQEBAQUAA4IBDwAwggEKAoIBAQDUXvZMJFOzpM5enek3exzMojDo6Vm2678hXmfKu/c
  [CA]6lcTC5/+Q7GZ34aTYXUDB9N1R0cygxrnCHBhVN6ZqZQ8Nyab+aRIt0pixwZaOxYAlzoF7A
  [CA]7au15yqT8EJbBDkRUWt4CNBUV6rJQe+DxglfjRP8oWwcCGI7NWnm/FI/F3IpAVMSthLx7g
  [CA]/B/muKA9WTTYiQo9g4T1TvV9gWJ4/04dVtMzfipPNHd16Rz0I/Pk/TlkVJKzfZl84Je0wS
  [CA]HIkHaTLkserONsCYTxqFd1luj1VT2O6ThMcpsdr6zX/USXx+tHAabcUA0x49kMY1PveH5D
  [CA]qf9AUSF7C/50xYd0IHz41AgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEF
  [CA]PELWkxRblScUr4R+HX0TKM9rM9vMA4GA1UdDwEB/wQEAwIChDANBgkqhkiG9w0BAQsFAAO
  [CA]CAQEADECVWKF7+OtKpJoXpv2CpbnLraoazNMJUJ+Y4CHpIJt/ikL8GT5psRL+zQ5089dMr
  [CA]VDPr4r3vmIq99NhvAnF7KUTGdSNqIQf7uBQffJqxfSNCrf0a6yitgDtCHEXxTNp7E7iesE
  [CA]HNuF26J31s/3iiEY9qjvxtiyPG5sCViLFS8Jt593oiwm5s2e1egaS4LeD75S8+zxoN+MVf
  [CA]AUmq9DF/XHrM4MurTbJ0A64rb/ngGMocFUh3ZvtbZt0ylgrQ6Se3lQWSqkmg2lbBpjWll7
  [CA]bvk3hkM6Tzms+BFBYgRADCRa5Npg7ojqhGc9nnX8EkuGaanmvsauK1i4r8MBAC4fdkQ=="
  [CA]],
  "alg": "RS256"
}

The representation of the keys depends on the type of key and signing algorithm. The above JWK represents an RSA signing key. Let’s break it down to understand what kind of information we have in the JWK:

kty: stands for key type and identifies the cryptographic algorithm family used with the key, such as RSA.

use: stands for public key used and indicates whether the public key is used for encrypting data or verifying the signature. For encryption, use’s value is enc, and for signature validation it’s sig.

n: the RSA key’s modulus, represented as the base64url-encoding of the integer in octets.

e: the RSA key’s public exponent, represented as the base64url-encoding of the integer in octets.

kid: stands for key ID and represents a unique identifier for the key.

x5t: stands for X.509 certificate SHA-1 thumbprint, also known as the certificate fingerprint. A certificate thumbprint is the base64url-encoding of the result of applying a hashing function, in this case SHA-1, over the X.509 certificate.

x5c: stands for X.509 certificate chain, and it represents the chain of certificates bound to the public key. Each certificate is base64url-encoded. The first element in the array represents the public key we must use to validate the token’s signature.

alg: stands for algorithm and identifies the algorithm used to sign the token using this key.

When you use RSA signing keys, IdPs give you the details of the signing key’s certificate, which contains the public key. The certificates are represented using the X.509 standard, which is a widely used format for representing RSA certificates. To learn more about the details included in the JWKS endpoint, check out RFC 7517 (“JSON Web Key (JWK)” by M. Jones, May 2015, https://datatracker.ietf.org/doc/html/rfc7517#section-4) and RFC 7518 (“JSON Web Algorithms (JWA)” by M. Jones, May 2015, https://www.rfc-editor.org/rfc/rfc7518.html).

Now let’s look at the contents of the JWT issued by Auth0 to understand how we use the information from the JWKS endpoint to validate the token. Here’s the header and payload of one such token issued from apithreats.com:

# Token header
{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "080E8CtRFAnnLlgK3dk8Y"
}
# Token payload
{
  "iss": "https://apithreats.eu.auth0.com/",
  "sub": "6faab069",
  "aud": "https://apithreats.com",
  "iat": 1732014181,
  "exp": 1732100581,
  "azp": "QDg7IrUH8jJ27ibGDRaEG5qW24dGOeWH"
}

From the header, we know that the ID of the key used to sign the token is 080E8CtRFAnnLlgK3dk8Y (kid field). We use this information to find the JWK necessary to validate the token’s signature. Depending on the JWT library we’re using, we’ll have different implementations for handling the JWKS and using them for validation. PyJWT provides a PyJWKClient class that loads the keys directly from the JWKS endpoint. Listing 8.11 gets an instance of that class and assigns it to a variable named jwks_client. PyJWKClient exposes various methods for finding the key, one of them named get_signing_key_from_jwt(), which listing 8.11 uses to retrieve the key. Finally, we invoke PyJWT’s decode() function passing it the token, the JWK object, the allowed signing algorithms, as well as the audience and the issuer we expect in the token’s payload.

Listing 8.11 Validating a JWT using keys from the JWKS endpoint

# file: ch08/api.py
import os
import jwt
[...]

jwks_endpoint = os.getenv("JWKS_ENDPOINT")    #A

assert jwks_endpoint is not None, "JWKS_ENDPOINT env variable needed"

jwks_client = jwt.PyJWKClient(jwks_endpoint)    #B

def validate_token(token, audience):    #C
    return jwt.decode(
        jwt=token,
        key=jwks_client.get_signing_key_from_jwt(
            token    #D
        ),
        algorithms=["RS256"],    #E
        audience=audience,
        issuer="<issuer>",
    )
[...]

The last line in the listing invokes the validate_token() function. To test the script, replace the <audience> placeholder with your own audience and the <issuer> placeholder with your own tenant’s URI. Then, paste an access token into the <token> placeholder and run the script. We are now in a position to issue and validate tokens from an IdP. The next step is to include the validate_token() function in our API to protect our endpoints from unauthorized access. We’ll do that with custom middleware in the next section!

8.5 Adding an authorization middleware

Web servers, including APIs, often have tasks that must be performed on every request. For example, as illustrated in figure 8.6, configuring headers correctly in every response, making sure they include the right CORS and Content-Security-Policy configuration. If the web server uses cookies, make sure cookies are correctly configured. And if it uses CSRF tokens, ensure they’re injected into every form. For APIs, a common task is validating request and response bodies, making sure their data conforms to the API schemas and data models. It would be cumbersome to add all these capabilities manually to every endpoint, and instead, web servers use the concept of middleware to define common tasks that will be applied to every request. One such common task is authorizing user access in every request, and therefore, in this section, we’ll add an authorization middleware to our API.

NOTE

A common question is whether we should validate access tokens in our API implementation code, or whether we should outsource such functionality to a component such as an API Gateway. As you’ll learn in section 9.1, API Gateways can validate access tokens and more, and if you have one, you should certainly use it for that purpose. However, bear in mind that API Gateways, like any other piece of software, can malfunction, and in that case you want to have a reliable fallback in your code to validate access tokens. Also, note that API Gateways may not be able to validate all access tokens, especially if yours have some customization. Also, some of your APIs may not be fronted by API Gateways. In such cases, add an authorization middleware to your applications as described in this section.

Figure 8.6 Processing a request involves a series of common tasks such as validating the headers, validating authentication, parsing and validating the request body, configuring CORS response headers, and so on. Since these are common tasks that must be applied to all requests, web servers normally implement them in the middleware.

Every web and API framework has its own way of creating custom middleware. The API under ch08/api.py in the code repository for this book is implemented with FastAPI, which allows us to add authentication middleware in various ways. One common way is creating a standard middleware class, and another way is using dependency injection. FastAPI’s dependency injection system ships with utilities that know how to handle bearer tokens, and they also automatically update the OpenAPI specification with the corresponding security schemes. Among other things, that means we’ll be able to keep using the Swagger UI under the /docs endpoint to access protected endpoints. For those reasons, we’ll use dependency injection to create our authorization middleware.

Listing 8.12 shows how we add authorization to the API using dependency injection. The code in listing 8.12 builds on top of listing 8.11, highlighting the new additions with bold case and omitting with ellipses the code we already explained earlier. The core of the implementation is the authorize_access() function, which we inject into every protected endpoint using FastAPI’s Depends() function (despite the CamelCase name, it’s a function). Depends() allows us to include custom processing steps in each request, such as applying the authorize_access() function.

authorize_access()takes a parameter named creds with a special type annotation:

Annotated[HTTPAuthorizationCredentials, Depends(HTTPBearer())]

Annotated is a special type annotation in Python that allows us to specify the parameter’s type and some metadata about it, using the following format: Annotated[<type>, <metadata>]. In our case, HTTPAuthorizationCredentials represents the type of creds, while the metadata, Depends(security), is another dependency injection that includes additional information about the credential’s format and how it should be handled. security is an instance of the HTTPBearer class, so it’s telling FastAPI that we are expecting a bearer token in the Authorization header. Using this metadata, FastAPI knows where to look for the bearer token (the Authorization header) and how to retrieve the token itself, by separating the Bearer prefix from the token itself. It then uses the token to create an instance of the HTTPAuthorizationCredentials class and returns it.

authorize_access () uses the validate_token() function we implemented in listing 8.12 to validate the token, and wraps the call within a try/except block to return an appropriate error response when token validation fails. If the token is valid, we return the token’s payload.

We apply authorize_access() to every endpoint we want to protect by declaring it as a dependency, as shown in the GET /protected-hello endpoint. GET /protected-hello’s router takes one parameter named user_claims, which represents the token’s payload returned by authorize_access(). This allows us to inspect the user claims to perform additional access control checks in our controller or our business layer. In this case, we use the sub field from the token’s payload to return a customized message to the user.

Listing 8.12 Authorizing requests in FastAPI using dependency injection

# file: ch08/api.py
import os
from typing import Annotated
import jwt
import requests
from fastapi import FastAPI, HTTPException, Depends
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from jwt import PyJWTError
from starlette.responses import RedirectResponse

[...]

def validate_token(token, audience):
    [...]

security = HTTPBearer()    #A

def authorize_access(
    creds: Annotated[    #B
        HTTPAuthorizationCredentials, Depends(security)
    ]
):
    try:    #C
        return validate_token(token=creds.credentials, audience=<audience>)
    except PyJWTError as error:    #D
        raise HTTPException(status_code=401, detail=str(error))

@server.get("/protected-hello")
def protected_hello(    #E
    user_claims: dict = Depends(authorize_access)
):
    return {    #F
        "message": f"hello {user_claims['sub']}"
    }

To try out the code from listing 8.12, run the server by executing the following command from the terminal: uvicorn api:server --reload. Navigate to the /docs page and hit the GET /protected-hello endpoint; you’ll receive a 401 (Unauthorized) status code response, with the message “Not authenticated”. To authenticate your requests, perform the login flow to obtain an authorization code and exchange it for an access token following the steps described in section 8.4.1. Then click on the Authorize button in the UI, paste the access token, and call the GET /protected-hello endpoint, as shown in figure 8.7. If you followed the steps correctly, this time you’ll obtain a successful response.

Figure 8.7 To authenticate requests in the Swagger UI, open the authorization panel, paste the token, close the panel, and execute the request.

Pause a moment to think about what we’ve achieved so far: we’ve integrated with a standard OpenID Connect provider, successfully issued access tokens, and added a robust authorization layer to our API. Everything we’ve done so far adds a great layer of security to our API, but there’s more we can do. In the remaining sections of this chapter, you’ll learn to authorize access controls based on user roles and to implement user-based access controls.

8.6 Implementing role-based access controls

Most APIs have a concept of use roles or groups, and as we discussed in section 7.7, this feature can be leveraged to harden the access controls in our API. For example, a ride-sharing application may distinguish between drivers and passengers, each with a distinct set of privileges and access to different endpoints. In this section, you learn to add role-based access controls to your APIs.

User roles must be configured on our IdP and assigned to each user. Every IdP has its own process for creating user roles, so please check out their documentation and follow the right steps. Appendix C contains a guide to creating permission sets and user roles, and how to assign them to user groups in Auth0. It also explains how to restrict access to certain APIs from specific clients. Follow the steps in Appendix C if you want to follow along with the examples in this section.

Appendix C creates an admin API, an admin client, and an admin user role. It also configures the Auth0 tenant to give exclusive access to the admin API from the admin client. In this section, we add an admin endpoint to the API under ch08/api.py and restrict its access to users with admin roles. The first step is to allow users to log in as admins. To log in as an admin, a user must have the admin role assigned to them, and they must request access to the admin API during the login process. To make this happen, we create a new login flow for admin users, as shown in listing 8.13. Listing 8.13 builds on top of the code we’ve written so far to ch08/api.py, with the previous code omitted with ellipses. Since the admin login flow uses a different client, we pull the admin client ID from the environment and make sure the value is set. Then we add the GET /admin/login endpoint, which is very similar to the GET /login endpoint from listing 8.9, with two important differences: this time we request access to the admin API through the audience parameter, and we use the admin client’s ID.

Listing 8.13 Adding an admin role-specific login flow

# file: ch08/api.py
[...]

admin_client_id = (    #A
    os.getenv("AUTH0_ADMIN_CLIENT_ID")
)
assert (
    admin_client_id is not None,
    "AUTH0_ADMIN_CLIENT_ID env variable needed."
)

@server.get("/admin/login")    #B
def admin_login():
    return RedirectResponse(    #C
        "<authorization_endpoint>"
        "?response_type=code"
        f"&client_id={admin_client_id}"
        "&redirect_uri=http://localhost:8000/docs"
        "&audience=<audience>"
    )

To run the code, replace the <authorization_endpoint> placeholder with your tenant’s authorization endpoint and the <audience> placeholder with your own audience. When the user logs in through the GET /admin/login endpoint, they’re redirected back to the /docs page with the authorization code in the URL. To exchange the authorization code for an access token, we add a new GET /admin/token endpoint that uses the client_secret from the admin client to perform the exchange, as shown in listing 8.14. We pull the admin client’s secret from the environment and then implement the GET /admin/token endpoint using the admin client’s credentials.

Listing 8.14 Exchanging the authorization code for an admin access token

# file: ch08/api.py
[...]

admin_client_secret = (    #A
    os.getenv("AUTH0_ADMIN_CLIENT_SECRET")
)
assert (
    admin_client_secret is not None,
    "AUTH0_ADMIN_CLIENT_SECRET env variable needed."
)

@server.get("/admin/token")    #B
def get_admin_access_token(code: str):
    payload = (    #C
        "grant_type=authorization_code"
        f"&client_id={admin_client_id}"
        f"&client_secret={admin_client_secret}"
        f"&code={code}"
        "&redirect_uri=http://localhost:8000/docs"
    )
    headers = {"content-type": "application/x-www-form-urlencoded"}
    response = requests.post(    #D
        "<token_endpoint>",
        payload, headers=headers
    )
    return response.json()    #E

To run the code, replace the <token_endpoint> placeholder with your tenant’s token endpoint. The admin login flow is complete. We can now log in as admins and obtain admin access tokens. Now we can add admin endpoints that are only accessible to admin users. Listing 8.15 adds one such endpoint, GET /admin/hello. To validate admin access, we also create a new authorize_admin_access() function with the same signature we introduced in listing 8.12, which allows FastAPI to process the credentials as a bearer token in the Authorization header. authorize_admin_access() checks that the audience in the token is correctly set to the expected value, and that the token’s payload contains a permissions array with the admin role in it. If those conditions aren’t met, we respond to the user with a 401 response. Finally, we add a GET /admin/hello endpoint with a dependency on the authorize_admin_access() function, which ensures only admin users can access it.

Listing 8.15 Restricting access to admin endpoints to admin users

# file: ch08/api.py
[...]

def authorize_admin_access(    #A
    creds: Annotated[HTTPAuthorizationCredentials, Depends(security)]
):
    try:
        claims = validate_token(
            creds.creds, audience="<audience>"
        )
    except PyJWTError as error:
        raise HTTPException(status_code=401, detail=str(error))
    if (    #B
        "admin" not in claims.get("permissions", []
    ):
        raise HTTPException(
            status_code=401, detail="This endpoint is only for admins"
        )
    return claims

@server.get("/admin/hello")
def admin_hello(    #C
    user_claims: dict = Depends(authorize_admin_access)
):
    return {"message": f"hello admin {user_claims['sub']}"}

To test the new additions to the API, replace the <audience> placeholder with your API’s audience. Then, set the required environment variables, AUTH0_ADMIN_CLIENT_ID and AUTH0_ADMIN_CLIENT_SECRET in your environment, and start up the server with the following command: uvicorn api:server --reload. Visit the admin login page on http://localhost:8000/admin/login in your browser and log in, then exchange the authorization token for an access token using the GET /admin/token endpoint. Authorize your requests following the steps in figure 8.6, and hit the GET /admin/hello endpoint.

We’ve now completed the journey to adding robust authentication and authorization to our API following well-established standards and best practices. Follow the steps outlined in this chapter to authenticate and authorize requests to your APIs, and keep the process lean and simple.

8.7 Summary

OpenAPI uses the concept of security schemes to document how users must authenticate their requests. A security scheme describes how users obtain access tokens and authenticate their requests, including details such as the authorization endpoint, the type of token and its placement, and the access scopes.

We can apply a security scheme to the whole API using the specification’s global security field, or we can apply it per operation, using the security field available under the URL path’s HTTP method.

To work with JSON Web Tokens (JWTs), we must use robust libraries that support a wide variety of signing algorithms, including PS256.

To sign JWTs using an asymmetric algorithm, we need a public and private key pair, which we can generate using OpenSSL’s genpkey command.

When validating JWTs, validate all the token’s claims and constrain the choice signing algorithm to prevent attacks that manipulate the token’s alg field.

When using an OpenID Connect (OIDC) provider, all the information we need to integrate with it is available under the discovery endpoint, available under the following URL path: /.well-known/openid-configuration.

To issue an access token through an OIDC provider, we redirect the user to the OIDC’s authorization endpoint. After successful login, the user is redirected back to our site with an authorization code, which we then exchange for the access token.

To validate access tokens issued by an identity provider, we use the public key corresponding to the private key used to sign the token. The list of public keys is available under the OIDC’s JWKS endpoint. We use the token header’s kid field to find the public key, and we pass the whole JWK object to the signature validation function.

To authorize a request, we check if the access token is valid. This is a task that must be performed on every request, and to accomplish that, it’s common practice to implement it in the middleware.

To restrict access to certain resources to specific groups of users, we use role-based access controls (RBAC). A common implementation of this is to include the user role in the access token’s payload, and to check that the token contains the right user group in the restricted operations.

9 Secure API infrastructure

This chapter covers

Improving API management with API gateways

Configuring secure network topologies

Protecting APIs from attacks against OSI model layers 3-6

Preventing application-level attacks with web application firewalls

Once we are ready to deploy our APIs, we have to think about how we protect the infrastructure they are going to operate in. In previous chapters, you’ve learned how to make APIs secure by design, how to implement robust authentication and authorization systems, and so on. These techniques help us prevent application-level attacks. As you’ll learn in this chapter, there are other forms of attacks that target lower levels of the networking stack, such as port scanning and denial of service attacks, which can compromise our security posture. You’ll learn about the types of vulnerabilities threat actors exploit to perform such attacks, and how you can harden your servers and use off-the-shelf solutions to protect your infrastructure.

You’ll learn about the impact of your network topology on your security posture. API infrastructure includes multiple elements, such as web servers, databases, and queues, and not all of them should be directly exposed to external users or allowed to receive external traffic. You’ll learn how to create restricted access to your API infrastructure with bastion servers, establish a demilitarized zone for your inbound traffic, and prevent lateral movement between our servers.

When it comes to managing and protecting APIs, two technologies help us improve our security posture with minimal effort: API gateways and web application firewalls (WAFs). We begin the chapter with an in-depth discussion of API gateways and how they help us manage our API inventory, control the attack surface, improve visibility and monitoring, and apply consistent access controls and secure configuration to all requests and responses. We’ll close the chapter with an overview of WAFs and how they secure our APIs, the types of exploits they protect against, and their limitations. As you’ll see, the technologies and solutions we discuss throughout this chapter can significantly improve your API security posture with minimal effort, but they are not silver bullets, and ultimately, the fate of our security relies on the correct application of all the practices and techniques we’ve discussed throughout the book.

9.1 API gateways

API gateways are tools that help us simplify the process of APIs governance. API governance involves publishing and retiring APIs, exposing their documentation, applying certain standards to API documentation, enforcing their usage policies, controlling access, monitoring them, and so on. As illustrated in figure 9.1, API gateways perform many of those functions by sitting in between the API client and the API servers. API gateways are particularly useful when we have multiple APIs, as they help us apply the same standards consistently across all our APIs.

DEFINITION

An API gateway, in the sense used in this chapter, is an API management solution software. However, the concept of API gateway originates in a pattern that helps us expose a single entry point for multiple backend APIs. To learn more about the API gateway pattern, see Chris Richardson’s Microservice Patterns (Manning, 2020), pp. 259–291; https://livebook.manning.com/book/microservices-patterns/chapter-8/point-8620-53-297-0). Modern products that implement the API gateway pattern bring additional features that help us simplify API governance, including authentication and authorization checks, monitoring, inventory management, schema validation, rate-limiting, etc.

Figure 9.1 API gateways serve as a single entry point to multiple backend APIs, helping us to provide a unified API style, manage our API inventory, apply consistent access controls and security configuration, and improve observability.

API gateways emerged as a pattern in microservices architecture to address the complexities of accessing multiple types of APIs across different endpoints. Originally, API gateways served three major goals:

Provide a single entry point for all services

Translate various API styles into a single style

Implement the backends-for-frontends pattern

Let’s break down each point. Providing a single entry point for all services means implementing service and API discovery. For example, figure 9.2 shows a microservices architecture for an e-commerce site. The backend consists of multiple APIs, each deployed across various computing choices, such as serverless, Kubernetes clusters, and individual servers. As shown in figure 9.2, without some intermediary layer that brings it all together, API clients must access every API on their own URIs.

This places a big responsibility on the API clients, who must know where and how to access each API. It also makes it more difficult to secure our platform, since we have multiple entry points with different infrastructure characteristics that we must secure and harden individually. It also means every API is responsible for implementing its own authentication and authorization controls, inventory management, and monitoring solutions. API gateways solve this problem by becoming the entry point to our platform and redirecting every request to its corresponding service.

Figure 9.2 Without an API gateway, API clients must know where and how to access each API. It also means every API must implement its own access controls, inventory management, and observability solutions.

The second use for API gateways is to translate protocols and API styles. For example, as illustrated in figure 9.3, the microservices in an e-commerce application may expose different types of APIs, such as REST, GraphQL, gRPC, and asynchronous APIs (AsyncAPI) using MQTT. In that scenario, it would be cumbersome for API clients to implement integrations with all those different API styles, so an API gateway works as a translation layer and exposes a unified API style, such as REST in figure 9.3.

Note

MQTT is a lightweight publish-subscribe messaging protocol for asynchronous communication between machines. It’s a very popular protocol for the Internet of Things (IoT), and for instance, AWS IoT platform is built on top of MQTT (https://docs.aws.amazon.com/iot/). MQTT originally stood for Message Queuing Telemetry Transport, but since 2013, the MQTT technical committee decided that it is not an acronym anymore (https://groups.oasis-open.org/higherlogic/ws/public/document?document_id=49028).

Figure 9.3 An API gateway can serve as a protocol and API style translation layer, helping us to provide a unified API style to our consumers. For example, our backend may consist of GraphQL, REST, and asynchronous APIs, and we can expose them all as a REST interface through an API gateway.

The third use for API gateways is to help us implement the backends-for-frontends (BFF) pattern. The idea is that the APIs that backend services expose aren’t necessarily tailored to the needs of API clients. For example, as shown in figure 9.4, a social media application may have two types of clients: mobile and browser-based for desktop. Due to the constraints of mobile applications, including limited storage and network connectivity, we want to minimize the number of requests we send to the server and the amount of data we retrieve. To accomplish that goal, we create an API gateway for the mobile application that aggregates data from multiple backend APIs, saving the mobile application multiple round trips to the server, and ensuring it gets only the data it needs. By filtering out the data that the client doesn’t need, the API gateway helps us address the problem of excessive data exposure that we discussed in section 4.4.2.

Figure 9.4 API gateways help us implement the backends-for-frontends pattern. The goal of BoF is to encapsulate the complexity of our backend APIs and expose interfaces that are tailored to the needs of our clients.

Modern API gateways can serve the functions we’ve discussed so far and more. In addition to the capabilities discussed above, most API gateways nowadays also support the following features:

Manage API Inventory

Enforce access controls

Validate request and response data

Apply security configuration in responses

Rate limiting requests

Collect request logs and produce usage analytics

Let’s break down each feature. Managing API inventory means helping us manage our API’s lifecycle and surface exposure. As we discussed in section 3.5, two major problems stemming from improper inventory management are shadow and zombie APIs. Zombie APIs are deprecated APIs that should have been retired but are still exposed. As our APIs evolve over time, we add new features, improve capabilities, and enhance their security. Often, these changes are backward-incompatible, which means we must release new versions of our APIs. As illustrated in figure 9.5, at any given point, we may have various versions of our APIs live at the same time, and it’s important to have a robust process for retiring old versions. Failing to do so means we end up with deprecated APIs running in production that are less secure and nobody monitors any longer - aka zombie APIs.

Figure 9.5 API gateways provide an effective solution to API inventory management. By channeling all inbound traffic through the API gateway, we take a declarative approach to which APIs and versions we expose, and we have up-to-date visibility of our attack surface.

API gateways help us mitigate the risk of zombie APIs by giving us visibility of all the API versions currently exposed, and allowing us to delete, disable, or sunset old versions. Some API gateways also allow us to configure an expiration date and include the sunset header in our responses automatically (RFC 8594, “The Sunset HTTP Header Field” by Eric Wilde, May 2019, https://datatracker.ietf.org/doc/html/rfc8594).

API gateways also help us manage the risk of shadow APIs. As we saw in section 3.5, shadow APIs are undocumented APIs or endpoints. As illustrated in figure 9.6, when developers are free to publish new endpoints and APIs without going through a planning and documentation process, those APIs often end up undocumented, untested, unprotected, and not being monitored, hence becoming a juicy target for threat actors.

Figure 9.6 API gateways give us control and visibility over the endpoints we expose, helping us tackle the problem of shadow endpoints, which are undocumented endpoints that are often untested and less protected.

API gateways help us mitigate the risk of shadow and zombie APIs by acting as the entry point to our platform. As illustrated in figure 9.6, the idea is to force all incoming traffic through the API gateway. To publish a new endpoint or a new API, we must register them first with the API gateway. Most API gateway products include dashboards where we can see and monitor all our APIs, which gives us instant visibility of our attack surface.

Moving down the list of features offered by modern API gateways, we have the ability to enforce access controls. As we saw in chapters 7 and 8, the first line of defense for APIs is authentication and authorization. As illustrated in figure 9.7, controlling access to an API follows a series of steps, the first being verifying that the request is correctly authenticated. After the authentication check, we also verify that the user belongs to the right user group, has the right permissions and/or scopes, and owns or otherwise has the right to access the requested resource. API gateways help with the first step in figure 9.7 – the authentication check. Many API gateways ship with out-of-the-box support for JWT validation, mutual TLS (mTLS, see section 7.5.1) authentication, and other forms of authentication.

Figure 9.7 API gateways help us apply consistent access controls following industry standards, providing out-of-the-box support for JWT validation, mTLS, and other forms of authentication.

API gateways support request and response data validation capabilities. This is a very powerful feature that ensures our APIs apply our security-by-design constraints. As we saw in chapter 6, we can address many security vulnerabilities from a design perspective, but those designs are of no use if we can’t enforce them. API gateways help us do that. Good API gateways support request and response data validation by simply feeding an API specification, and can apply our design constraints on request and response bodies, URL parameters, and headers. Once this configuration is in place, the API gateway will reject requests that don’t conform to our schemas, and it will raise appropriate errors when our responses are malformed, too.

API gateways also help us apply secure configuration in our responses. As we saw in section 5.3, security misconfigurations are pervasive and a major source of API exploits. As illustrated in figure 9.8, we can configure API gateways to automatically inject secure header configuration in our responses, such as CORS (Access-Control-Allow-Origin), HSTS (Strict-Transport-Security), content security policy (Content-Security-Policy), and others. This type of configuration is particularly important if our APIs are consumed by a browser-based application.

Figure 9.8 API gateways help us consistently apply secure headers configuration to our responses, ensuring they contain the right CORS policy, HSTS configuration, content security policy, etc.

Finally, modern API gateways also feature customizable rate-limiting capabilities that help us restrain excessive calls from our consumers, as illustrated in figure 9.9. Further, since API gateways serve as the entry point to our platform and have access to all request logs, many of them provide out-of-the-box observability and usage analytics.

Figure 9.9 API gateways often offer out-of-the-box rate-limiting capabilities, observability, and analytics. These capabilities help us manage misbehaving API clients and draw intelligence about the usage patterns of our APIs.

API gateways are an invaluable component in complex API platforms, and it’s worth spending the time to research your options and choose the right one for your use cases. Daniel Bryant, author of Mastering API Architecture (O’Reilly, 2022), views the process of choosing an API gateway as a type 1 decision, meaning it’s a one-way, irreversible choice. The reason is that API gateways are sticky products.[1] As you’ve learned in this chapter, API gateways offer many useful features, and once you put an API gateway in front of your system, your operations become tightly coupled to it, so it’s important to make a good choice from the start.

Note

The concept of type 1 decisions goes back to a letter to shareholders written in 2015 by Jeff Bezos, founder and CEO of Amazon (https://s2.q4cdn.com/299287126/files/doc_financials/annual/2015-Letter-to-Shareholders.PDF). In that letter, he distinguished between type 1 and type 2 decisions. Type 1 (or one-way) decisions are irreversible, for example, choosing a cloud provider or a programming language – once we make that choice, it’s very difficult to change it. Type 2 (or two-way) decisions are reversible and are the type of decisions we make on a daily basis, for example, naming a class or a function, or choosing one design pattern over another.

The best way to choose the right API gateway for your platform is by checking out a few options, their features, and testing their capabilities. If you use a cloud provider like AWS, Azure, or GCP, it makes sense to consider their API gateway offerings since they integrate well with the rest of their services. AWS offers AWS API Gateway (https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html), Azure offers API Management (https://learn.microsoft.com/en-us/azure/api-management/), and GCP offers Apigee (https://cloud.google.com/apigee). Check out also popular API gateway vendors like Kong, Traefik Labs, Tyk, Axway, MuleSoft, solo.io, KrakenD, Gravitee, Apinto, and Apache APISIX

Now that we understand the crucial contribution that API gateways make to our security posture, let’s dive into other aspects of API architecture, beginning with network topology, which is the topic of the next section.

9.2 Secure network topologies

APIs run on a network, and the security configuration of that network has a direct impact on our security posture. As we saw in section 3.5, network misconfiguration is a major source of security breaches, so it’s worth spending the time to evaluate the security of our network configuration and what we can do to improve it. Network security includes everything from secure network topologies to firewall configuration, DDoS protection, and more. The point of this section is not to learn everything there is to know about network security, but to understand how network topology affects our API security posture. As an API architect, you may not implement the security measures discussed in this section yourself, but you must be able to understand how your network topology impacts your threat models and what to do about that.

Network topology refers to the structure of our network. As illustrated in figure 9.10, a common strategy in network security is to partition our network into multiple segments or subnets. Each segment represents a small network with varying degrees of access. Some segments may have access to the Internet while others are completely isolated. Many cloud providers call subnets with Internet access public subnets, and subnets without Internet access private subnets. Services running within the same network segment can talk to each other, and to enable communication across segments, we must create a dedicated network bridge. We call the subnet exposed to the Internet the demilitarized zone (DMZ) of our network. We don’t deploy services directly to the DMZ; instead, the DMZ acts as an Internet gateway to our services via an API gateway, a reverse proxy, a load balancer, and similar technologies.

DEFINITION

In networking, demilitarized zone (DMZ), also known as perimeter network, is the segment of our network that is exposed to all traffic on the Internet. We call it the DMZ in reference to the border between North Korea and South Korea, since it represents a buffer zone between untrusted and potentially malicious traffic and our internal network. The DMZ consists of highly hardened components designed to protect our system, such as API gateways, load balancers, and web application firewalls (WAFs).

Figure 9.10 The demilitarized zone (DMZ) is a region of our network that is exposed to the Internet. It’s good practice to channel inbound traffic through the DMZ and keep all our services in private subnets. The DMZ can include elements such as an API gateway and a bastion server.

Network segmentation in the cloud

If you’re building in the cloud, you can achieve network segmentation in different ways. For example, in AWS, you can provision different virtual private clouds (VPCs) for different types of resources. AWS VPCs are isolate network environments, so by default the services deployed to one VPC can’t communicate with those running in another VPC. You can enable communication between VPCs through VPC peering if needed though.

Within the same VPC, you can create public and private subnets. By default, services deployed to different subnets within the same VPC can talk to each other, but you can prevent that by restricting inbound traffic through the subnet’s access control list (ACL). You can apply additional inbound traffic restrictions by configuring AWS security groups. Security groups work like application-level firewalls, and allow you to restrict or allow traffic by IP and port.

To learn more about network segmentation in AWS and how to make your architecture compliant with strict security requirements such as the payment card industry data security standard (PCI DSS), check out the whitepaper “Architecting for PCI DSS Scoping and Segmentation on AWS,” by Ted Tanner, Abdul Javid, and Padmakar Bhosale, November 2024 revision (https://d1.awsstatic.com/whitepapers/compliance/architecting-pci-dss-segmentation-scoping-aws.pdf).

Network segmentation allows us to divide our network into subnets with characteristics suited to each component in our platform. For example, databases are highly sensitive components. A database must be accessible to our services, but there’s no reason to expose it to the Internet. Therefore, as shown in figure 9.10, it makes sense to deploy our database to private subnets. To enable external access to the database, we typically use a bastion server that tunnels the connection from our local machine to the database. The bastion server sits in a network with Internet access and a bridge that gives access to the private network where the database lives.

DEFINITION

A bastion server or bastion host is a specially hardened server that allows access to our internal networks through the DMZ. Typically, the bastion server only allows inbound traffic through port 22 for SSH connections for a small allowlist of trusted IPs. Since bastion servers allow us to jump from one network to another, they are also known as jump boxes.

Some applications distinguish between public and private or internal APIs. Internal APIs tend to handle highly sensitive data and are usually less protected. As we discussed in section 3.4, we should treat internal APIs as if they were public and protect them as such. As illustrated in figure 9.11, an additional measure of security is to deploy them within a private subnet, so that external access to those APIs is disabled by design. If we need external access to those APIs, we can enable it through a bastion server.

Figure 9.11 A bastion server is a server that gives us access to an internal network. We use bastion servers to access internal or private APIs and databases. The bastion server sits in the DMZ and it’s hardened to allow traffic only from certain origins.

Public APIs must allow inbound traffic from the Internet, and as illustrated in figure 9.12, this is typically accomplished by deploying them to an internal network with a reverse proxy, such as an API gateway, forwarding traffic from the DMZ to the internal network. This brings all the benefits of using the API gateway as the entry point to our platform, including visibility of our attack surface, enforcement of access controls, and so on.

DEFINITION

A reverse proxy is a surrogate server that sits in front of our website and forwards client requests to our servers. Reverse proxies can be used for multiple purposes, including SSL termination (aka encrypting and/or decrypting SSL-encrypted traffic), blacklisting malicious IPs, load balancing requests across multiple backend nodes, and more.

This approach follows the principles of zero-trust architecture (see section 3.2) - it makes our APIs inaccessible by default unless we declare the endpoints through the API gateway. If one of our services needs to expose a new endpoint, it must be declared through the API gateway, which means our attack surface is always visible and up to date, and all endpoints receive the same security protections configured in the API gateway. As shown in figure 9.12, this approach also means that our services can expose additional endpoints that are only available internally for testing or to support back-office functionality.

Figure 9.12 Using a combination of API gateway and bastion server, we make sure our APIs only expose well-documented, tested, and secure endpoints to the Internet, while keeping internal endpoints or endpoints in development only accessible to internal users and developers.

Network segmentation is also useful to isolate services with more sensitive data and operations. This is also known as preventing lateral movement between networks. For example, a healthcare application may have a service that manages patients’ appointments and another service that manages their medical records. The medical records service is more sensitive than the appointments management service, hence we deploy it to its own network where we can apply stricter access controls and enhanced monitoring for compliance with healthcare regulations, such as HIPAA. Microsegmentation allows us to de-scope the appointments management service from compliance audits since it is completely isolated from the medical records service and lateral movement isn’t possible between the two, i.e. a user within the appointments management service has no way of accessing data from the medical records service.

Now that we know how to architect our network to protect our most sensitive components and user data from unauthorized access, let’s take a look at the type of attacks threat actors can carry out at the lowest levels of the networking stack and how to mitigate them.

9.3 Protecting against layers 3-6 attacks

Before a malicious actor can send a SQL injection attack to our APIs, they have to reach our servers, open a connection, maintain it, and be able to send correctly formed packets of data. All of this happens in the network, transport, session, and presentation layers of the open systems interconnection (OSI) model, also known as layers 3, 4, 5, and 6. Threat actors have every opportunity to exploit vulnerabilities in those layers to compromise the security posture of our APIs. In this section, we discuss the exploits available against OSI layers 3-6 and what we can do to mitigate them. The goal is to get you acquainted with vulnerabilities in this area that may affect your API security posture and what mitigations you can implement to contain such risks.

Figure 9.13 The open systems interconnection (OSI) model breaks down the elements required to enable communication between systems into seven layers. Layer 1 describes the physical medium, such as switches and cables, and layer 7 the application layer, such as APIs. Many API attacks target layers 3-6 and require special security measures.

The OSI model is an abstraction that describes the elements required to enable communication between systems. As illustrated in figure 9.13, the OSI model breaks down the inter-system communication process into 7 layers:

Physical layer: the physical medium over which communication happens, such as cables.

Data link layer: the delivery of data frames between nodes in the local network using media access control (MAC) addresses. It detects and corrects transmission errors. An example of layer 2 protocol is the Ethernet.

Network layer: the delivery of data packets over a network using IP addresses.

Transport layer: the transmission of data between two points in a network. The supported features depend on the implementation and may include acknowledgment, flow control, and multiplexing. Popular implementations of layer 4 are the user datagram protocol (UDP) and the transmission control protocol (TCP).

Session layer: manages the continuous exchange of data between two nodes over a network, providing mechanisms to establish, maintain, and close the connection.

Presentation layer: also known as the syntax layer, it takes care of translating the data packets exchanged over the network into data that can be consumed by applications, and encoding application data into a format suitable for transmission. An example of this is the serialization and deserialization of JSON data over the network.

Application layer: this is the layer at which applications use the semantics of the data exchanged over the network to communicate with each other. Examples of application layer implementations are HTTP, XMPP, FTP, SSH, TLS/SSL, MQTT, etc.

Protocols like HTTP and XMPP depend on the reliable transmission of data over a network (layers 3 and 4) for an extended period of time (layer 5), and the translation of such data into messages that they can interpret (layer 6). APIs build on top of layer 7. Throughout the book, we’ve seen strategies to protect our APIs against attacks such as scalping, fuzzing attacks, mass assignment, SQL injection, and so on. We call these attacks application-level or layer 7 attacks. In addition to layer 7, layers 3-6 can also expose vulnerabilities that threat actors may exploit to compromise the security posture of our APIs. In this section, we study examples of those vulnerabilities and how to architect our system to mitigate them.

Denial of service attacks

Denial of service (DoS) is one of the most common types of attacks against APIs, often referred to as distributed denial of service (DDoS). We say distributed when the attack is launched from a distributed network of nodes or origins. The goal of a denial of service attack is to bring down our services, and this can be accomplished with a handful of well-crafted malicious requests or by overwhelming our servers with millions of concurrent requests. To crash our servers using a handful of requests, threat actors exploit vulnerabilities in the application layer. For example, if our application exposes a GET /products endpoint with no bounds on how many products we can retrieve at once, threat actors can overwhelm our system by requesting millions of items in each request (see section 5.1, where we discuss this type of vulnerability in greater detail). Alternatively, threat actors can attempt to overwhelm our system by sending millions of requests, also known as volumetric DoS attacks. Volumetric DoS attacks can target the application layer, for example by sending millions of HTTP requests, but more commonly, they target layers 3-6 of the OSI model. Examples of layers 3-6 attacks are ICMP, SYN, and UDP flood, DNS and UDP amplification attacks, IP fragmentation attacks, etc.

To learn more about DDoS attacks, check out Chad Kime’s “Complete Guide to the Types of DDoS Attacks”, eSecurity Planet, December 19 2022 (https://www.esecurityplanet.com/networks/types-of-ddos-attacks). Eric Chou and Rich Groves’s Distributed Denial of Service (DDoS) (O’Reilly, 2018), and Suzanne Aldrich’s “Anatomy of DDoS”, presented at Builderscon Tokyo 2017 (https://www.slideshare.net/slideshow/anatomy-of-ddos/78561801).

One of the most common types of attacks against layers 3-6 is denial of service (DoS), often referred to as distributed denial of service (DDoS) when the attack is launched from a distributed network of nodes. The goal of a DDoS is to bring down our services - see sidebar “Denial of Service Attacks” for more details. Threat actors employ various techniques to launch a DDoS attack against our servers. The most common types of DDoS attacks are volumetric attacks that target layers 3 and 4. Volumetric attacks flood the server with a high volume of requests. For example, a common exploit is flooding the victim’s servers with messages following the Internet control message protocol (ICMP), such as ping or echo requests. As illustrated in figure 9.14, the idea is to overwhelm the server with ICMP packets without waiting for their replies, thereby consuming server resources and making it unable to respond to legit user requests. ICMP flood is an example of layer 3 attack and can be prevented by disabling ICMP requests.

Figure 9.14 A common type of denial of service (DoS) attack is ping flood. The goal of this attack is to overwhelm the server with an unbound number of requests until it no longer can respond to request from legitimate users.

The evolution of the HTTP protocol sometimes introduces new exploits for DDoS attacks. For example, HTTP/2 (RFC 9113 by M. Thompson and C. Benfield, June 2022, https://datatracker.ietf.org/doc/html/rfc9113) includes several features that improve web performance. However, those features can also be exploited for malicious purposes. One such exploit is HTTP/2 rapid reset. Let’s see how it works.

As illustrated in figure 9.15, HTTP/2 allows clients to serialize their requests into individual streams, a feature known as HTTP/2 multiplexing. Each stream has an identifier, and the client can send multiple streams over the same connection. All the streams can go over the same TCP connection, allowing clients to send multiple concurrent requests over the same socket. Servers allow a maximum number of concurrent streams, typically 100. To make efficient use of the streams, HTTP/2 allows clients to close a stream if it deems it’s no longer necessary. For example, when a user visits a website, the browser serializes into various streams the requests for all the assets, files, and content it needs to load the page. As the user scrolls down the page, some of those images and text may fall out of the viewport (i.e. out of view) before they’re loaded, and the client may decide to reset and cancel those streams so that it can open new ones and fetch new resources.

Figure 9.15 HTTP/2 uses the concept of streams to allow clients to serialize multiple requests over a single TCP connection. Each stream has an identifier, and if the client deems a certain request no longer needed, it can cancel it with a reset stream request.

If used as intended, HTTP/2 reset is a powerful feature that allows clients and servers to make efficient use of resources. However, this feature can also be abused to launch denial of service attacks. To accomplish that, a threat actor configures a client to send the maximum number of streams allowed by the server, and to close the streams right away, reusing the available streams to send additional requests followed by their corresponding resets. This process is repeated until the server resources are exhausted by requests. By canceling the streams, the client never exceeds the allowed maximum number of concurrent streams and therefore has an opportunity to flood the server with an overwhelming number of requests. Although not confirmed, research by Cloudflare indicates that HTTP/3 may also be vulnerable to rapid reset exploits (https://www.cloudflare.com/the-net/rapid-reset-ddos/).

One of the biggest DDoS attacks in history: exploiting HTTP/2 rapid reset

In August 2023, HTTP/2 rapid reset led to one of the biggest DDoS attacks in history. The attack targeted major cloud providers Cloudflare, Google Cloud, and AWS, and it reached up to 400 million requests per second.

For details on the attack and vendors dealt with it, check out Renato Losio, “Cloudflare, Google and AWS Disclose HTTP/2 Zero-Day Vulnerability,” InfoQ, November 5, 2023 (https://www.infoq.com/news/2023/11/http2-rapid-reset-vulnerability/), Juho Snellman and Daniele Iamartino, “How it works: The novel HTTP/2 ‘Rapid Reset’ DDoS attack,” Google Cloud Blog, October 10, 2023 (https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack), and Lucas Pardue and Julien Desgats, “HTTP/2 Rapid Reset: deconstructing the record-breaking attack,” The Cloudflare Blog, October 10, 2023 (https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/).

Rapid reset is one of several exploits available in HTTP/2. To learn about additional exploits, check out Nafeez, “On the recent HTTP/2 DoS attacks,” The Cloudflare Blog, August 13, 2019 (https://blog.cloudflare.com/on-the-recent-http-2-dos-attacks/).

Another common exploit against layers 3-6 is TCP/UDP port scanning. Port scanning alone doesn’t hurt our servers, but the knowledge threat actors gain from it allows them to create tailored attacks against the open ports. As shown in figure 9.16, servers have 65535 usable port numbers, and many popular services run by default on specific ports. For example, PostgreSQL runs by default on port 5432, MySQL on port 3306, SSH on port 22, and Jenkins on port 8080. By discovering which ports are open on our servers, threat actors can launch specific attacks tailored to the services running on those ports. If a threat actor discovers an unauthenticated Jenkins server running on port 8080, they will gain access to our internal network and sensitive configuration details of our system.

Figure 9.16 Port scanning is an attack that aims to discover open ports in our server. Threat actors use this information to tailor attacks to the services running on each port. For example, in this graphic, a threat actor discovers an unprotected Jenkins running on port 8080 and hijacks it to gain access to our system.

Many DDoS attacks require hitting the server with multiple requests per second. A common mitigation strategy against such attacks is to track the IP where the requests originate and block or blacklist it. This strategy is effective when the malicious requests come consistently from the same IP or set of IPs. As illustrated in figure 9.17, to bypass such protection, threat actors spoof or mask their IPs. IP spoofing is the practice of modifying the IP header in a packet with other IP values, while IP masking hides the origin IP behind other servers such VPNs and TOR nodes. These techniques make it look like the requests are all coming from different sources, making it more difficult for the receiving server to identify the DDoS attack and block the requests.

Figure 9.17 IP masking is the practice of crafting TCP packets with origin IPs different from our own. This technique helps threat actors bypass anti-DoS protection measures that focus on tracking the origin IP of the requests.

How do we ensure our APIs are adequately protected against layers 3-6 attacks? If your APIs run on servers that you must manage and configure, such as on-premises servers, EC2 instances on AWS, or droplets on DigitalOcean, the first course of action is to harden those servers by closing unnecessary ports, disabling or removing unnecessary protocols and services, removing or restricting privileged users or groups, creating strict firewall rules, and securing remote access endpoints. If you are in this situation, you want to hire specialized consultants to audit your server security configuration.

Hardening your servers protects them against unauthorized access. How do we go about protecting our APIs against DDoS, port scanning, and other kinds of malicious traffic? A good solution is using products specialized in this type of protection. For example, organizations like Cloudflare, F5, Fastly, and Akamai, offer popular and very effective products in this space. Make sure you learn about their features and capabilities, and evaluate how well they integrate with your own infrastructure before making your choice. A special type of product that offers protection against malicious traffic across all the networking layers of the OSI model, including layer 7, is the web application firewall (WAF). In the next section, we learn more about WAFs and how they help us secure our APIs.

9.4 Fending off malicious traffic with web application firewalls

Web application firewalls (WAFs) are a type of firewall that protects websites against application-level attacks. As illustrated in figure 9.18, WAFs are typically deployed as a reverse proxy for our APIs. A reverse proxy is a surrogate server that sits in front of our website and forwards client requests to our servers and helps us add an additional layer of protection (as in the case of the WAF), SSL termination (aka encrypting and/or decrypting SSL-encrypted traffic), or load balance requests across multiple backend nodes.

DEFINITION

A web application firewall (WAF) is a firewall that protects websites from application-level attacks such as injection and cross-site scripting (XSS) attacks. Modern WAFs offer more comprehensive security, including protection against zero-day vulnerabilities and layers 3-6 attacks.

Figure 9.18 A web application firewall (WAF) is a firewall that protects against application-level attacks such as SQL injection and cross-site scripting (XSS). In this example, a WAF rejects a request from a threat actor that contains a SQL injection attack.

The first generation of WAFs was designed to protect against well-known application vulnerabilities, such as injection and cross-site scripting (XSS) attacks. They do this by inspecting the request contents and assessing the probability that they represent a malicious attack. For example, the following request represents a SQL injection attack:

GET /payments?filter=' or 1=1--

The above line represents a GET request against a /payments endpoint which sets the filter query parameter to a SQL statement. As illustrated in figure 9.19, if the API is vulnerable to SQL injection, filter’s SQL statement will be executed as part of the endpoint’s database query.

Figure 9.19 If our API is vulnerable to SQL injection, threat actors can exploit this vulnerability to cause unexpected behavior in our system or disrupt our services. In this example, a threat actor disables the user_id constraint on the SQL query, gaining access to records from other users.

WAFs protect our applications by identifying the SQL injection statement and either blocking the request or cleaning up the malicious content from the request. If the WAF cleans up input instead of blocking the request, it can still become vulnerable to WAF bypass attacks. For example, the following request can trick some WAF implementations into cleaning the input into an executable SQL statement:

GET /payments?filter='+o//r+1=1--

Some WAF implementations use matching patterns and rules to identify malicious content, while others use more sophisticated probabilistic methods. Matching patterns look for keywords or sequences of keywords that look like database injection statements or malicious scripts. There’s a tradeoff in how strict the pattern-matching rules are. On one hand, if the list of patterns isn’t comprehensive enough, threat actors will be able to construct injection statements that bypass the WAF’s analysis engine. On the other hand, if the list of patterns is too comprehensive and strict, the WAF may block legit requests that happen to look like an injection attack. For example, a blog post explaining how injection works will contain examples of SQL injection and other malicious statements, and such content may look suspicious to a very strict WAF.

A probabilistic WAF analyzes the request’s content and assigns a score to it. For example, the request we saw at the beginning of this section, GET /payments?filter=' or 1=1--, is likely to receive a high probability of SQL injection attack score, while the blog post explaining how injection attacks work will receive a lower probability score. The WAF’s analysis engine reaches such determination by looking at the context in which the SQL statement appears.

Nowadays WAFs include API-specific support like schema validation. If we supply the WAF with our API specification, it will ensure that the request payloads conform with our data schemas and otherwise reject them with a useful error message and status code.

Modern WAFs also offer more exhaustive security coverage that goes beyond application-level attacks. For example, most modern WAFs include protection against layers 3-6 attacks such as those we discussed in section 9.3. WAFs also often protect against zero-day vulnerabilities, which are vulnerabilities for which there’s no known fix. This contrasts with the workflow of ethical hackers and cybersecurity researchers, who work closely with organizations and technology vendors to ensure the vulnerabilities are fixed before disclosing them to the public.

DEFINITION

Zero-day vulnerabilities are vulnerabilities for which there is no known fix. We say zero-day since organizations have zero days left to fix the vulnerability since the time of their discovery.

One of the most infamous examples of zero-day vulnerabilities is HeartBleed (https://heartbleed.com/). Disclosed in April 2014, HeartBleed is a vulnerability in the OpenSSL library that allows threat actors to leak content in memory handled by the OpenSSL library from the server, such as traffic data between users and the server. This allows threat actors to retrieve sensitive data like usernames and passwords to hijack user accounts, and even get access to the secret key of the X.509 certificate that is used to encrypt traffic and identify the server. HeartBleed was disclosed on April 1, 2014 and a fix became available on April 7, 2014. As organizations struggled to update their systems on time, many became victims of data breaches, including hospitals, Canada’s tax authority, UK’s parenting social network Mumsnet, and others (“US hospital hack 'exploited Heartbleed flaw',” BBC News, August 20, 2014, https://www.bbc.com/news/technology-28867113). As of 2024, it is estimated that around 60,000 servers around the world are still vulnerable to HeartBleed (Andrada Fiscutean, “Heartbleed: When Is It Good to Name a Vulnerability?” DarkReading, May 14, 2024, https://www.darkreading.com/vulnerabilities-threats/heartbleed-when-is-it-good-to-name-a-vulnerability).

A more recent example of zero-day vulnerability is HTTP/2 rapid reset, which we discussed at length in section 9.3. WAFs protect our applications from zero-day vulnerabilities by implementing mitigating countermeasures at runtime. Such mitigations are especially effective against layers 3-6 attacks. However, as we saw earlier in this section, detecting and mitigating application-level attacks, such as SQL injection, is a more challenging task. WAFs are wonderful tools as our first line of defense, but there is only so much a WAF can do, and we shouldn’t mistake having a WAF for being protected against all kinds of API vulnerabilities. In fact, overconfidence in WAFs capabilities is a major factor leading organizations to relax their cybersecurity hardening process and therefore publishing more vulnerable APIs (“Overconfidence in API security posture leaves enterprises at high risk,” Security Magazine, June 21, 2022, https://www.securitymagazine.com/articles/97860-overconfidence-in-api-security-posture-leaves-enterprises-at-high-risk).

9.5 Summary

API gateway is a pattern from microservices architecture that allows us to provide a single entry point for multiple backend APIs and offer a unified API style to our users.

Modern API gateways help us manage our API inventory, monitor our APIs, consistently enforce access controls, and apply secure configuration.

Network segmentation is the practice of dividing our network into isolated subnets. Segmentation allows us to tailor each subnet to the security requirements of each API.

The demilitarized zone (DMZ) is a subnet in our network that is exposed to the Internet. It’s good practice to channel all inbound traffic through the DMZ and keep all our services in subnets without Internet access.

The open systems interconnection (OSI) model is an abstraction that breaks down the elements required to enable communication between systems into seven layers. Layer 7 is the application layer, and the other layers describe lower-level elements of the networking stack. Many API attacks target OSI layers 3-6, including denial of service (DoS) and port scanning attacks.

Preventing layers 3-6 attacks requires hardening servers with strict firewall configuration, closing unnecessary ports, and other measures. It’s also good practice to use off-the-shelf anti-DoS protection services.

A web application firewall (WAF) is a firewall that protects against application-level attacks such as SQL injection and cross-site scripting (XSS) attacks. Modern WAFs also support data validation for APIs and protect against zero-day vulnerabilities and layers 3-6 attacks.

[1] Daniel Bryant, “The Busy Platform Engineer’s Guide to API Gateways,” APIs: Possibilities and Pitfalls organized by Jean Yang and hosted by O’Reilly, Oct 11, 2023 (https://learning.oreilly.com/live-events/apis-possibilities-and-pitfalls/0636920097623/).

10 Financial-grade APIs

This chapter covers

Open banking for the financial services industry

Delivering highly secure APIs with FAPI

Securing the authorization process

Adding non-repudiation capabilities to APIs

In recent years, we have witnessed a revolution in the financial technology (fintech) space. We have seen the emergence of organizations like Plaid, Bud, TrueLayer, Yappily, Yodlee, Stripe, and others, that provide financial market infrastructure. Financial market infrastructure is a critical component of today’s economy that allows businesses to easily manage financial transactions via APIs. We have seen the emergence of so-called NeoBanks or challenger banks, which are banks that operate exclusively online. We have seen businesses provide banking-like services without having a banking license.

All of this is possible thanks to financial APIs. Organizations that otherwise wouldn’t be able to process online payments due to the strict regulatory requirements can do that by using payment processors like Stripe and GoCardless. Payment processors take care of building a secure financial infrastructure that connects directly with banks to manage payments and other financial transactions, including refunds, chargebacks, and so on. The processor exposes an API that we use to set up a payment, and they process the operation by leveraging their direct integrations with the banks.

On the consumer side, we have seen the emergence of applications like Moneyhub, SlimPay, Plum, Emma, and others, that allow us to manage multiple bank accounts from a single place (also known as account aggregators), track our expenses, plan our savings, make payments, set up investments, and more, without having to log in to our bank account.

The secret sauce behind this fintech revolution is open banking, also known as open finance. Traditionally, banks have been closed organizations that want to own directly the relationship with their customers. However, a major regulatory push started by the European Union in 2015 forced banks to open up their data and capabilities through APIs in the interest of fostering financial innovation. Other countries like Brazil, Australia, and Japan, have implemented similar requirements, and, as of 2025, the United States has a proposal on the table to follow suit under section 1033 of the Consumer Act.

Open banking offers new business opportunities for financial institutions, but it also brings new risks and security challenges. Systems that were never designed to be exposed to third parties must now be accessible through APIs, with all the security concerns that come with it. The risks range from unauthorized access to sensitive data to account takeover and fraudulent operations. In this case, people’s livelihoods, homes, and entire life savings are at stake, so it’s crucial to implement the highest security standards for financial APIs. The OpenID Foundation (https://openid.net/) created a working group called FAPI to offer guidelines and meet the security requirements of financial-grade APIs.

In this chapter, you learn what those security requirements are, how they work, and how they protect against specific threat models. As you will see, these security measures are applicable to a wide variety of sectors beyond finance, and by applying them correctly, we can make the Internet more secure. If you work with applications in finance, law, healthcare, government, defense, and other highly sensitive fields, this chapter is very much relevant to you and you should make an effort to implement the security standards discussed here.

10.1 What is open banking?

Open banking is the practice of securely opening APIs to allow third-party applications to access customer bank accounts with user consent. Open banking allows developers to build applications that help customers securely manage their bank accounts from a single place without having to log in to their bank account. This ecosystem fosters innovation in the fintech space by allowing developers to build applications in niche areas like personal budgeting, expense tracking, accounting, and more, without having to worry about creating the financial infrastructure needed to perform financial transactions.

Note.

Open banking (with lower case) refers to the practice of exposing banking data and capabilities through APIs. It is not to be confused with Open Banking (with capital case), which is a non-profit organization in the UK pushing the standards for open banking (https://www.openbanking.org.uk/).

Open banking received a major push from the European Union in 2015, when the Council for the European Union approved the Revised Payment Services Directive, known as PSD2. PSD2 went into full effect on 14 September 2019 with the goal of improving the security of online payments and fostering competition and innovation within the payments industry. To encourage such competition, PSD2 requires financial institutions to share their data and payment capabilities through APIs. Many other countries have followed the trail of the EU, including Australia, Brazil, and Japan, with many other countries currently working out open banking mandates for their legislative frameworks, including the United States (see https://truelayer.com/reports/open-banking-guide/open-banking-around-the-world/ for an overview of open banking around the world and https://www.consumerfinance.gov/about-us/newsroom/cfpb-finalizes-personal-financial-data-rights-rule-to-boost-competition-protect-privacy-and-give-families-more-choice-in-financial-services/ for the current state of open banking in the US, as well as Chris Wood’s excellent overview of section 1033 in “What Does Section 1033 Mean for Open Banking in the US?,” Nordic APIs Blog, January 14, 2025, https://nordicapis.com/what-does-section-1033-mean-for-open-banking-in-the-us/).

DEFINITION.

Screen scraping is the practice of capturing user credentials to log in to their bank account and fetching their details or performing operations on their behalf using web scraping techniques. This practice is highly insecure since it involves handing over highly sensitive credentials to a third-party application and give it unlimited access to customer data.

Open banking changes the game for fintech developers. As illustrated in figure 10.1, before the advent of financial APIs, fintech applications would request users to share their login credentials and use them to log in to their bank accounts, then scrape their data from the bank’s website, a practice known as screen scraping (https://www.pymnts.com/bank-regulation/2024/journey-from-screen-scraping-to-open-banking-will-be-more-marathon-than-sprint/). Screen scraping is a perfect recipe for a security disaster: user credentials are passed around and stored in third-party applications, and those applications get unfettered access to user data and operations.

Figure 10.1 With screen-scraping, bank customers share their account credentials with a third-party application, who uses the credentials to log in to the bank’s website impersonating the customer.

Screen-scraping is also a classic example of an automated web attack, where threat actors use customer credentials and scrape their private and highly sensitive data. Allowing screen-scraping means banks have no way of distinguishing legitimate from malicious scraping, and by default, allow automated attacks on their websites. For all those reasons, open banking is a very welcomed move within the financial industry to provide a more secure and robust way of accessing customer accounts. To provide that level of security, the OpenID Foundation’s FAPI working group has been working on a series of documents that detail the specific threat models open baking APIs should protect against and how to do it. In the next section, we learn what FAPI is before jumping straight into the specifications.

10.2 What is FAPI?

FAPI is a working group within the OpenID Foundation dedicated to creating standards for APIs with high-security requirements. Originally, FAPI was an acronym for financial-grade APIs, and the goal of the working group was to respond to provide implementation guidelines to meet the security requirements of open banking APIs. FAPI has gone through two versions. FAPI 1.0 focused on the security requirements of financial APIs, while FAPI 2.0 is broader in scope and applicable in other industries like healthcare, and it brings additional security requirements.

Note.

FAPI 2.0 supersedes FAPI 1.0, however, you may encounter applications that still operate under the FAPI 1.0 model. If you need to learn how FAPI 1.0 works, check out FAPI 1.0’s baseline profile (https://openid.net/specs/openid-financial-api-part-1-1_0.html) and the advanced profile (https://openid.net/specs/openid-financial-api-part-2-1_0.html). The baseline profile defines a secure profile to protect operations that read sensitive data from a resource server, such as retrieving an account statement. The advanced profile introduces non-repudiation to protect operations that modify sensitive data in the resource server, for example, initiating a payment.

FAPI 2.0 consists of two profiles: the security profile (https://openid.bitbucket.io/fapi/fapi-security-profile-2_0.html) and message signing (https://openid.bitbucket.io/fapi/fapi-2_0-message-signing.html). The profiles are designed for interoperability among open baking providers and they build on top of OAuth 2.0 (RFC 6749, https://www.rfc-editor.org/info/rfc6749), OAuth 2.0 best current practices (RFC 9700, https://datatracker.ietf.org/doc/html/rfc9700), and the OIDC protocol (https://openid.net/specs/openid-connect-core-1_0.html). FAPI makes an opinionated and constrained use of these standards to procure a more secure way of authenticating users and authorizing their access to an API.

FAPI’s security guidelines have been adopted by UK’s Open Banking standards, Brazil’s Open Banking Initiative, Australia’s Consumer Data Rights Standards, and more. You can check the latest state of FAPI’s work in its Bitbucket repository: https://bitbucket.org/openid/fapi/src/master/. In the rest of this chapter, we dive straight into FAPI’s profiles, beginning with an analysis of its attacker model, which lays out the security requirements that FAPI addresses.

10.3 Understanding FAPI’s attacker model

FAPI 2.0 is designed to protect users and businesses from a specific threat model outlined in “FAPI 2.0 Attacker Model,” authored by D. Fett (December 2022, https://openid.net/specs/fapi-2_0-attacker-model-ID2.html). The attacker model does not describe specific attacks and instead aims to be a high-level reference for all possible types of threats that FAPI applications must successfully mitigate. The model breaks down its goals into three categories: authorization, authentication, and session integrity. Let’s discuss each goal.

The first goal of FAPI 2.0’s attacker model is authorization. As illustrated in figure 10.2, the idea is to ensure that users can only access resources they are allowed to. For example, if I make a payment using a payments application, I’m the only user allowed to inspect or modify the details of the payment, and the application must successfully implement that type of access restriction. Since authorization is carried by access tokens, the objective of the attacker model is to ensure that users can only obtain and utilize their own access tokens. In other words, the goal is to prevent threat actors from landing their hands in other users’ access tokens and using them to access their data.

Figure 10.2 FAPI’s threat model accounts for situations in which threat actors steal other users’ access tokens, and therefore FAPI applications must implement mechanisms to block the use of stolen tokens.

The second goal is authentication. As illustrated in figure 10.3, the idea is that users should not be able to log in using the identity of another user. For example, if I have an account in a healthcare application, no other user must be able to log in to my account. FAPI’s authentication model builds on top of OIDC and hence user identity is carried by ID tokens. Therefore, FAPI must satisfy the authentication goal by ensuring that ID tokens can only be bound to their legitimate owners.

Figure 10.3 FAPI accounts for scenarios in which threat actors attempt to assume the identity of another user, and therefore applications must be able to successfully prevent such attacks.

The third goal is session integrity. The idea is to ensure all messages exchanged during the authorization process maintain their integrity, which means threat actors can’t tamper messages with malicious data. For example, as illustrated in figure 10.4, a common exploit in OAuth is cross-site request forgery (CSRF) attacks against the authorization endpoint to steal user credentials. The attack works when a user is already logged in with their identity provider. In this scenario, a threat actor tricks the user into visiting a malicious website, where the threat actor sends an authorization request to the user’s identity provider. Since the user is already logged in, the identity provider automatically responds with an authorization code, which the malicious actor exchanges for an access token. The traditional way to protect against such attacks in OAuth is by using a state parameter in the authorization request. FAPI brings in additional security requirements to guarantee the integrity of user sessions.

Figure 10.4 A threat actor tricks a user into visiting a malicious website, which redirects the user to their identity provider via an authorization request, resulting in the leak of an authorization code that the malicious site then exchanges for an access token.

Now that we understand FAPI 2.0’s security goals, let’s see how we achieve them.

10.4 Securing APIs with FAPI 2.0’s security profile

The FAPI 2.0 framework is a collection of specifications that describe how our applications must work in order to achieve a high level of security. The two main documents that describe FAPI 2.0’s requirements are the security profile (“FAPI 2.0 Security Profile,” by D. Fett, D. Tonge, and J. Heenan, draft 4 as of February 12, 2025, https://openid.bitbucket.io/fapi/fapi-security-profile-2_0.html) and the message signing profile (“FAPI 2.0 Message Signing,” by D. Tonge, D. Fett, and J. Heenan, draft as of February 12, 2025, https://openid.bitbucket.io/fapi/fapi-2_0-message-signing.html). The security profile describes the baseline security requirements of FAPI 2.0 applications, while the message signing profile describes more advanced features for a higher level of security aimed at critically sensitive operations and applications. In this section, we discuss FAPI 2.0’s security profile.

Figure 10.5 The main agents involved in FAPI exchanges are the resource owner, the user agent, the client, the authorization server, and the resource server.

FAPI 2.0’s security profile describes a secure process for issuing access tokens and using them to access sensitive data from a resource server. As illustrated in figure 10.5, the main agents in the FAPI 2.0 framework are:

The client: the application that accesses protected data on behalf of the user.

The resource owner: the user who owns data in the resource server.

The user agent: the browser or mobile application.

The authorization server: the server that identifies the user and issues access tokens.

The resource server: the server that holds user-sensitive data.

FAPI 2.0 addresses the security concerns described in the attacker model we analyzed in section 10.3, and to that end it brings a number of constraints and requirements on how authorization servers, resource servers, and clients must operate. Let’s break down the security requirements of the FAPI 2.0 framework.

As we saw in section 10.3, FAPI 2.0 aims to deliver strong authentication and authorization processes. Threat actors must not be able to take on other user accounts, steal their access tokens, or access their data. To achieve those goals, FAPI 2.0 requires that all OAuth clients be confidential clients. As illustrated in figure 10.6, confidential clients run in a server, hence their code is not exposed, and therefore can store credentials with which they authenticate their requests to the authorization server. If the client exposes a web application that runs in the browser (the user agent), all interactions with the authorization server are done directly by the server. Meanwhile, public clients, by virtue of being public, have their code exposed to the public, hence they can’t store credentials and therefore can’t authenticate their requests with the authorization server. As shown in figure 10.6, an example of public client is a client that runs in the browser (the user agent). In FAPI 2.0, clients must be confidential and manage interactions with the authorization server to produce authorization request URLs and securely obtain access, refresh, and ID tokens through a back channel.

Figure 10.6 Public clients run in the user agent, and their code is exposed, therefore they can’t store credentials to authenticate with the authorization server. Confidential clients run in a server and their code isn’t exposed, therefore they can store credentials with which they authenticate with the authorization server.

Clients can be authenticated using mutual authentication (mTLS, see section 7.5 for more details on this) or by signing requests with a signing key issued exclusively for the client. The latter method is known as private_key_jwt, and it requires the client to issue and sign a new JWT on every request to prove its authenticity.

A common source of exploits in OAuth authorization requests is the fact that authorization redirect URLs contain the authorization parameters in the URL, such as the client_id and the redirect_uri. For example, a typical request URL may look like /authorize?response_type=code&client_id=s6BhdRkqt3&state=xyz&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb. The problem with this approach is that it exposes sensitive parameters through the URL, which may be visible to threat actors eavesdropping on the network, and they leak into network logs.

Exposing authorization request parameters through the URL also allows threat actors to forge malicious authorization request URLs designed to steal user credentials, as we illustrated in figure 10.4. To prevent this type of exploits, FAPI 2.0 requires clients to use JWT-secured authorization requests (JARs) in combination with the pushed authorization requests (PARs) protocol. The basic idea behind JARs and PARs is that the OAuth client wraps the authorization request parameters in a JSON Web Token, as illustrated in figure 10.7. The client sends the signed JWT to the authorization server over an authenticated request in exchange for a unique URL that allows the user agent to initiate the authorization process securely. See section 10.5 for an in-depth explanation of the JAR and PAR protocols.

Figure 10.7 Using PAR, the client exchanges the authorization request parameters with the authorization server for a unique request URI. The client then uses the request URI to form the authorization request URL and redirects the user agent to it.

To further prevent the problem of threat actors crafting malicious authorization requests, FAPI 2.0 forbids the use of open redirect URLs. Open redirect is a feature that allows client applications to set the value of the redirect_uri in the authorization URL dynamically. Instead of that, FAPI 2.0 requires clients to register their allowed list of redirect URLs, and authorization servers must verify that the request_uri included in the authorization request matches one of the pre-registered URLs.

The requirements discussed so far ensure our applications can authenticate users and issue access tokens securely. The next step is to ensure that threat actors can’t use other users’ access tokens even if they happen to land their hands on them. To this end, FAPI 2.0 requires applications to use sender-constrained tokens. As we discussed in section 7.5, sender-constrained tokens are tokens that contain proof that the token belongs to their legitimate user, and we achieve this by binding the token to the user’s TLS certificate (mTLS for certificate-bound tokens) or with a special JWT signed with a user-exclusive key, called DPoP token.

FAPI 2.0 also discourages the use of refresh token rotation. Since only confidential clients are allowed and their interactions with the authorization server must be authenticated over a secure backchannel, refresh tokens don’t bring additional benefits and their usage brings unnecessary complexity and potentially degrades user experience.

Finally, FAPI 2.0 brings some constraints on the algorithms we can use to sign access tokens. A common source of vulnerabilities in APIs that use JSON Web Tokens (JWTs) to authorize access revolves around the token signature. As we learned in chapters 7 and 8, JWTs indicate which algorithm was used to produce the signature through the alg field in the token’s header. JWTs carry the alg field to support cases where tokens may be signed with different algorithms. When our API supports multiple types of signing algorithms, we use the alg header to select the right algorithm to validate the token’s signature. As we saw in section 8.3, threat actors abuse this feature to trick APIs into accepting forged tokens by exploiting a vulnerability named algorithm confusion. In section 8.2, we also learned about factors that can make the RS256 signing algorithm vulnerable to Bleichenbacher’s ’06 signature forgery attack. To prevent all these problems, FAPI 2.0 only allows signing JWTs with PS256, ES256, and EdDSA (with the Ed25519 variant) algorithms. Resource servers must reject tokens signed with any other algorithm. See section 8.3 for techniques to constrain the allowed signing algorithms when validating JWTs.

Now that we understand the security requirements of FAPI 2.0’s security profile, let’s dive into the details of how JAR and PAR work.

10.5 Securing authorization requests

To access protected data in a resource server, we must obtain an access token. As we saw in section 7.3 and illustrated in figure 10.8, we begin the process to obtain an access token with an authorization request.

Figure 10.8 OAuth flows begin with an authorization request, during which the resource owner verifies their identity and consents to the client to access their data in the resource server. The authorization server responds with an authorization code which the client exchanges for an access token.

The authorization request must contain certain parameters, such the response_type, the client_id, and the redirect_uri, which are needed to authenticate the user correctly and redirect them back to the application. For example, a typical authorization request may look like this:

GET /authorize?response_type=code&client_id=s6BhdRkqt3&state=xyz&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb HTTP/1.1
Host: server.example.com

The problem with this type of authorization request is that it includes all the client data in the URL and is not authenticated. As illustrated in figure 10.9, including sensitive data in the URL is dangerous since it may become available to threat actors who manage to eavesdrop the connection between the user and the authorization server. This vulnerability may allow threat actors to perform a man-in-the-middle attack and tamper the authorization request, for example, by overriding the redirect_uri and setting its value to a malicious site where they may seize control of the authorization code. As we saw in section 10.3, exposing authorization parameters in the URL also allows threat actors to forge authorization requests for CSRF attacks against the token endpoint. Sensitive data included in the authorization request may also leak in request logs, hence becoming visible to anyone with access to the logs. Finally, since the authorization request doesn’t contain a proof of identity, the authorization server doesn’t have the necessary information to discriminate between legitimate and forged authorization requests.

Figure 10.9 Exposing authorization request parameters in the URL means threat actors can forge authorization requests to steal access tokens from other users.

To protect authorization requests from the exploits illustrated in figure 10.9, we use JWT-secured authorization requests (JARs). JARs were introduced in RFC 9101 (“The OAuth 2.0 Authorization Framework: JWT-Secured Authorization Request (JAR)”, by N. Sakimura, J. Bradley, and M. Jone, 2021, https://www.rfc-editor.org/rfc/rfc9101.html).

DEFINITION.

JSON-secured authorization request (JAR) is a method for protecting authorization requests from tampering, forgery, hijacking, and other exploits by including sensitive values in the request in a special JSON Web Token known as the request object. The request object is signed using a private key assigned to the client to prove its identity.

As illustrated in figure 10.10, a JAR is an authorization request that allows clients to include the authorization request parameters in a special JSON Web Token (JWT) known as the request object.

Figure 10.10 With JAR, the client wraps the authorization request parameters in a signed JWT and uses it to produce the authorization request URL. It then redirects the client to the authorization server using that URL.

As illustrated in listing 10.1, a request object contains all the standard claims of the JWT specification plus the parameters required in the authorization request, such as client_id, redirect_uri, and response_type. The token may also contain fields typical of an authorization request, such as scope, and cross-site request forgery (CSRF) protection parameters, such as state and nonce. If you need a refresher on the meaning and use of these authorization parameters, please refer back to section 7.4.

As you can see in listing 10.1, client_id and iss have the same value, since the request object JWT is issued by the client. Also, the aud claim represents the authorization server since the JWT is intended for the authorization server. The request object is signed with a private key assigned to the client, which is how we prove that the request comes from the right client and has not been tampered with, and base64url encoded for safe transmission over HTTP. If the request object contains sensitive information that you don’t want to expose, you can also encrypt it using JSON web encryption (JWE, check out the sidebar “Help! I don’t want to expose my API’s JWT claims!” in chapter 7 for more information on JWEs).

Listing 10.1 Request object payload for a JWT-secured authorization request

  {
   "iss": "s6BhdRkqt3",
   "aud": "https://auth.apithreats.com",
   "response_type": "code id_token",
   "client_id": "s6BhdRkqt3",
   "redirect_uri": "https://client.example.org/cb",
   "scope": "openid",
   "state": "af0ifjsldkj",
   "nonce": "n-0S6_WzA2Mj",
   "max_age": 86400
  }

Once the client has produced the signed request object, we must send it to the authorization server in the authorization request. We have two ways of passing the request object to the authorization server: pass by value and pass by reference. Let’s take a close look at each option.

As illustrated in figure 10.11, passing by value means the client sends the request object directly to the authorization server. In this case, we base64url encode the request object JWT in the authorization request through a query parameter named request.

Figure 10.11 When we pass the request object by value, the client produces the authorization request JWT and inserts its value in the request’s request parameter.

As illustrated in figure 10.12, passing by reference means we include an identifier that references the request object in a query parameter named request_uri in the authorization request. Since we are only passing an identifier, the actual request object must be stored either in the client or in the authorization server. Figure 10.12 illustrates what happens when we store the request object in the client’s backend. In this case, we have to make the request object available to the authorization server so it can verify its authenticity when the user agent sends the authorization request. We make the request object accessible by creating a unique URL accessible through a GET request. This URL is the request object’s identifier, it’s short-lived (less than a minute), and valid only for a one-time access. When the authorization server receives the authorization request with the request object’s reference in the request_uri parameter, it verifies its legitimacy by accessing its value on the provided URL.

Figure 10.12 When we pass the request object by reference, the client stores the request object in its database and issues a URL that identifies the object. It then inserts the URL in the authorization request, in the request_uri parameter. The request URI is accessible to the authorization server to verify that the authorization request is legitimate.

Figure 10.13 illustrates the authorization flow when we store the request object in the authorization server. In this case, the client pushes (i.e. sends) the request object to the authorization server, which returns a request object URI. The URI is a random identifier, since it only needs to be accessible to the authorization server, and it’s short-lived.

Figure 10.13 The client can send the request object directly to the authorization server in exchange for a request URI, which the client inserts into the authorization request’s request_uri parameter.

In addition to the request or the request_uri parameters, the authorization request must also include the client_id to identify the client. Below is an example of an authorization request URL using the pass-by-value method for sending the request object:

https://server.example.com/authorize?client_id=s6BhdRkqt3&request=eyJhbGciOiJSUzI1NiIsImtpZCI6ImsyYmRjIn0.ewogICAgImlzcyI6ICJzNkJoZFJrcXQzIiwKICAgICJhdWQiOiAiaHR0cHM6Ly9zZXJ2ZXIuZXhhbXBsZS5jb20iLAogICAgInJlc3BvbnNlX3R5cGUiOiAiY29kZSBpZF90b2tlbiIsCiAgICAiY2xpZW50X2lkIjogInM2QmhkUmtxdDMiLAogICAgInJlZGlyZWN0X3VyaSI6ICJodHRwczovL2NsaWVudC5leGFtcGxlLm9yZy9jYiIsCiAgICAic2NvcGUiOiAib3BlbmlkIiwKICAgICJzdGF0ZSI6ICJhZjBpZmpzbGRraiIsCiAgICAibm9uY2UiOiAibi0wUzZfV3pBMk1qIiwKICAgICJtYXhfYWdlIjogODY0MDAKfQ.Nsxa_18VUElVaPjqW_ToI1yrEJ67BgKb5xsuZRVqzGkfKrOIX7BCx0biSxYGmjK9KJPctH1OC0iQJwXu5YVY-vnW0_PLJb1C2HG-ztVzcnKZC2gE4i0vgQcpkUOCpW3SEYXnyWnKzuKzqSb1wAZALo5f89B_p6QA6j6JwBSRvdVsDPdulW8lKxGTbH82czCaQ50rLAg3EYLYaCb4ik4I1zGXE4fvim9FIMs8OCMmzwIB5S-ujFfzwFjoyuPEV4hJnoVUmXR_W9typPf846lGwA8h9G9oNTIuX8Ft2jfpnZdFmLg3_wr3Wa5q3a-lfbgF3S9H_8nN3j1i7tLR_5Nz-g

JARs allow us to protect the integrity of our authorization requests by including the signature element of the JWT in the request object. This ensures that threat actors can’t tamper authorization requests by including malicious redirect URIs and other values. However, being able to send the request object by value means we are still exposing its values through the wire. If the signing process applied to the request object isn’t robust enough, threat actors may be able to break the signing key and forge new tokens. To prevent this type of scenario, RFC 9126 (“OAuth 2.0 Pushed Authorization Requests,” by T. Lodderstedt, B. Campbell, N. Sakimura, D. Tonge, and F. Skokan, Sept 2021, https://www.rfc-editor.org/rfc/rfc9126.html) introduces the concept of pushed authorization requests (PARs).

Pushed authorization requests include an additional layer of security by requiring the client to send or push the request object to the authorization server in return for a URI. Authorization servers that support PAR requests are required to include a PAR endpoint to which the client sends the request object for registration. The PAR endpoint must be documented in the server metadata or discovery endpoint.

As illustrated in figure 10.14, in a pushed authorization request, the client issues the request object and pushes it to the authorization server in exchange for a request URI. There are two variants of these flow:

The client sends the authorization request parameters to the authorization server’s PAR endpoint in the body of a POST request.

The client produces the signed request object and sends it to the authorization server’s PAR endpoint in the body of a POST request.

We use a POST request in either case to prevent leakage of the authorization request parameters in the URL. The client must include authentication details when sending the pushed authorization request, and the authorization server must check the client’s credentials before processing the pushed authorization request’s payload. This ensures that the authorization server only processes legitimate requests.

Figure 10.14 In a pushed authorization request (PAR), the client issues the authorization request JWT and pushes it to the authorization server in exchange for a request URI, which the client then inserts into the authorization request’s request_uri parameter.

If the pushed authorization request is successful, the authorization server responds with a request object URI, which is for one-time use only and valid for a limited period of time (between 5 and 600 seconds). The user agent includes the request object URI in the authorization request in a parameter named request_uri, just like the case of JARs.

By combining JARs with PARs we leverage the benefits of client authentication to legitimize our authorization requests, and we mitigate the risk of authorization request tampering or forgery by removing the parameters from the URL.

Now that we understand how JARs and PARs work, we have all the ingredients necessary to implement FAPI 2.0’s security profile. The next step in our journey is FAPI 2.0’s message signing profile, which brings an additional layer of security for critically sensitive APIs.

10.6 Message signing

Message signing is an advanced FAPI security that builds on top of FAPI 2.0. The goal of the message signing profile is to allow APIs to feature non-repudiation capabilities. Non-repudiation refers to the ability to prove that a certain user undertook a specific action. This feature is important for applications in finance, healthcare, legal, and other fields in which user actions carry financial or legal risk, or applications in which user actions cannot be easily reverted. In all those cases, we must be able to prove that the user did indeed commit a specific action.

For example, on an e-commerce site, user Bob may have placed an online order for a non-refundable product. The item is delivered and Bob tries the product, then decides he doesn’t like it. However, since the product is non-refundable, he can’t get his money back, so he raises a claim stating that the order was placed by an online hacker.

How do we prove whether Bob did or did not place the online order? Following the security practices we’ve discussed so far in this book, we can ensure that the process of placing an order online is secure – free from business flow exploits, command and database injections, and so on. But we don’t have the right tools to prove whether Bob did really place the order or not. All we know is that an order was placed using Bob’s account. To achieve a higher level of confidence in the relationship between users and their actions, we need an additional layer of security, and that’s where message signing comes in.

Message signing achieves non-repudiation through signature proofs. The idea is to be able to prove that the authorization process and the resulting access tokens and their usage are all bound to their legitimate user, and we accomplish that by signing every message involved in this process.

The first step is to prove the authenticity of the authorization request. We achieve this by using pushed authorization requests (PARs). As we learned in section 10.5, PARs require the client to push the authorization request parameters to the authorization server in return for a request_uri. The client’s PAR request is authenticated, and the request_uri is short-lived and one-time use only, thereby mitigating the risk of relay and replay attacks.

If the authorization request is successful, the authorization server responds with an authorization code using the JWT-secured authorization response mode protocol (JARM). With JARM, the authorization server returns the authorization code within a JWT. The JWT’s signature proves that the response came from the legitimate authorization server. The goal of JARM is to protect the client from attacks where a threat actor may be able to intercept the response from the authorization server and modify its content before relaying it back to the user agent. In that scenario, the threat actor may be able to trick the client into sending the authorization code to a threat actor’s malicious server, thereby allowing the malicious actor to exchange the code for an access token. You can learn more about JARM from “JWT Secured Authorization Response Mode for OAuth 2.0 (JARM),” by T. Lodderstedt and B. Campbell (November 2022, https://openid.net/specs/oauth-v2-jarm-final.html).

In some cases, resource servers need to obtain additional information about an access token. The OAuth specification isn’t prescriptive about the type of token that can be used, and many implementations choose to use opaque tokens. Contrary to JWTs, opaque tokens can’t be inspected, and in such cases, resource servers need a mechanism to prove the token’s authenticity, i.e. to confirm it was issued by the right authorization server. The token introspection endpoint serves that need.

Authorization servers expose a token introspection endpoint to allow resource servers to obtain additional details about an access token. This functionality is also for JWTs, for example if the JWT has a unique identifier (the jti claim) and the resource server needs to confirm that the jti is legitimate. FAPI 2.0’s message signing profile requires that the token introspection endpoint be authenticated, and that the authorization server’s responses include the results of the introspection within a signed JWT. By requiring credentials, we mitigate the risk of threat actors forging requests to leak token data from our servers, and by signing the responses, resource servers can rest assured that the token introspection response is legitimate.

DEFINITION.

Token introspection is an endpoint in authorization servers that allows clients to obtain additional information about an access token. The requirements for token introspection endpoints are outlined in RFC 7662 (“OAuth 2.0 Token Introspection,” by J. Richer, October 2015, https://www.rfc-editor.org/rfc/rfc7662.html). RFC 9701 brings a layer of security for introspection endpoints by requiring them to wrap their responses within a signed JWT (“JSON Web Token (JWT) Response for OAuth Token Introspection,” by T. Lodderstedt and V. Dzhuvinov, January 2025, https://www.rfc-editor.org/rfc/rfc9701.html).

The final requirement of the message signing profile is that authorization servers sign ID tokens. ID tokens are tokens that contain identifying information about the user, and they’re issued after a successful authentication process with an OpenID Connect provider. As we saw in section 7.2, ID tokens have JWT format, including a header, a payload, and a signature. Although identity providers can issue ID tokens without signature, it’s best and common practice to issue signed ID tokens. The signature allows client applications to verify the authenticity and integrity of the token and hence trust the identity information contained in the token.

10.7 Summary

FAPI is a highly secure Open Authorization profile. While FAPI 1.0 was originally designed for financial-grade APIs, FAPI 2.0 aims to be useful for all APIs with strong security requirements.

FAPI 2.0’s attacker model satisfies three goals:

Authorization: every operation can only be performed by authorized parties.

Authentication: threat actors can’t impersonate other users.

Session integrity: threat actors can’t tamper, replay, or otherwise interfere with legitimate user requests.

FAPI only supports confidential clients to ensure interactions between the client and the authorization server are authenticated.

JWT-secured authorization requests (JARs) allow us to send authorization request parameters securely by wrapping them within a JWT known as the request object. The JWT guarantees the integrity of the payload, thereby mitigating the risk of tampering.

Pushed authorization requests (PARs) add a layer of security on top of JAR by requiring clients to push the request object directly to the authorization server. The PAR request must be authenticated to prove that the request is legitimate.

JWT-secured authorization response mode (JARM) requires authorization servers to return the output of a successful authorization request wrapped in a JWT. The JWT’s signature guarantees the integrity of the payload and therefore mitigates the risk of tampering.

Message signing a FAPI 2.0 profile that allows APIs to support non-repudiation. Non-repudiation is the ability to prove that a certain user undertook a specific action, and it’s crucial for applications in which user actions carry a legal or financial risk.

Message signing achieves non-repudiation with the use of signatures, which guarantee the integrity of the payloads and prove that they have a legitimate source of origin:

Signing authorization requests using PAR.

Signing authorization code responses using JARM.

Wrapping the response from the token introspection endpoint within a signed JWT.

Signing ID tokens.

11 Observability for API security

This chapter covers

What observability is and how it helps us protect our APIs

Leveraging logs, traces, and metrics for API observability and security

Instrumenting APIs to produce logs, traces, and metrics

Leveraging observability to detect threats and identify malicious actors

You’ve built your shiny new API and released it to the world. Developers are excited, and tons of new users sign up to use it and integrate it within their own applications. Life is good. Within a few days, though, your customers start complaining. Your API doesn’t always work as expected – sometimes, it returns malformed responses. Often, the API is down. And some users are reporting that their data is suddenly gone or appears to have been accessed and modified without authorization. How do you get a picture of what is going on, pin down the problems, and identify the root causes?

The answer to these questions is API observability. API observability is the practice of generating, collecting, and continuously analyzing data from our APIs. Without observability, it’s hard to tell how users engage with our APIs, trace errors, detect malicious activity, discover undocumented attack surface, and identify threats. And because of that, threat actors can sneak under the radar and attack your APIs without your knowledge. Lack of observability is a threat actor’s treasure trove.

In section 2.1, we learned about the NIST CSF’s five functions: identify, protect, detect, respond, and recover. In previous chapters, you learned to identify sources of risk to your APIs and to protect them against them. In this chapter, we tackle the detect function. It’s all good that we built a robust and secure API, but there’s no such thing as a perfectly secure system, and threat actors will always find a way through your APIs. In the world of cybersecurity, we say that a breach is not a matter of if, but when.

Good detection mechanisms are crucial for effective response to and recovery from a threat event, and the key to effective detection is observability. Observability allows us to understand how our users interact with our APIs and where our APIs are erroring or failing to behave as expected. Without observability, we don’t know when, where, and how errors happen, and worst of all, we don’t know if we ever are under attack. In this chapter, we’ll make sure none of that happens to our APIs!

11.1 What is API observability

Once our APIs are running in production, we need to know whether they are operating as expected, or whether they are erroring, degrading user experience, or otherwise causing unwanted outcomes. The traditional answer to identifying issues in production is monitoring. Monitoring is the practice of capturing data for predefined metrics, such as requests per second, response latencies, CPU usage, and such, and creating alerts for when the metrics cross a certain threshold.

Traditional monitoring excels at well-defined problems for which we can easily create metrics. But what about open-ended problems for which there is no clear metric? For example, how do we identify what happened to those users who claim their data was accessed and modified by unauthorized parties? For that type of analysis, we need a more comprehensive approach to application monitoring that captures richer details and puts them together in a way that makes it easy to run exploratory analyses on them. That’s where observability comes in.

Observability is the ability to measure the internal states of a system based on its outputs. The concept borrows from control theory, where the goal is to develop models that can drive a system into a desired state based on inputs, and the state of the system is determined based on its outputs. As illustrated in figure 11.1, when applied to software, the goal of observability is to determine the state of our systems based on their outputs or signals, such as logs.

Figure 11.1 By continuously analyzing our logs, we gain an understanding of the state of our system. In this example, we find that one endpoint is encountering an unusual error rate.

We call the outputs produced by software applications telemetry data. The most common type of telemetry data is logs, but it also includes things like metrics and other types of data that we’ll discuss later in this chapter. Telemetry data allows us to understand the state of our applications, their health, and the way users interact with them. All this information helps us reduce the mean time to recovery (MTTR), and it is crucial for ensuring the reliability and well-functioning of our APIs and services.

DEFINITION

Mean time to recovery (MTTR) is the time it takes to restore a service after a production failure such as an unexpected outage. MTTR is one of the key metrics used in the State of DevOps report by the DevOps Research and Assessment (DORA) team (the term used in the latest reports is “Failed deployment recovery time”), with top performing teams achieving an MTTR of less than an hour (Accelerate State of DevOps 2024, p. 13 https://dora.dev/research/2024/).

DEFINITION

Audit trails are chronological records of events of who did what, where, and when. We use audit trails to support cybersecurity investigations for accountability and regulatory compliance. This chapter describes how to produce application-level audit trails, but you should keep detailed logs of overall system and infrastructure changes too. If you deploy to cloud providers, you can use services such as AWS CloudTrail to keep a record of all actions performed on your infrastructure.

From a security perspective, observability also helps us detect threats and malicious behavior. In regulated environments, the information gathered through observability serves as audit trails and it is crucial to support security investigations. In the following sections of this chapter, you’ll learn to get your APIs ready for observability and to leverage observability for security.

11.2 Logs, traces, and metrics

In software, our systems output logs, traces, and metrics. These outputs are known as signals in OpenTelemetry (OTel), and they are the elements that help us understand and diagnose our systems. As Cindy Sridharan, author of Distributed Systems Observability (O’Reilly, 2018), puts it, logs, metrics, and traces are the pillars of observability. Let’s take a closer look at these elements and understand their uses.

DEFINITION

OpenTelemetry (OTel) is a standard observability framework (https://opentelemetry.io/). OTel is open source (https://github.com/open-telemetry), and it’s both vendor- and tool-agnostic. OTel defines a collection of APIs that observability tools must implement to be OTel-compliant, and it maintains a list of recommended SDKs (https://opentelemetry.io/docs/languages/) that we can use to instrument our applications to generate, collect, and export telemetry data. To learn more about OTel, check out Daniel Gomez Blanco’s Practical OpenTelemetry (Apress, 2023) and Michal Hausenblas’s Cloud Observability in Action (Manning, 2023). For a more practical approach, check out Phil Wilkins’s Logs and Telemetry (Manning, 2024), and for a generic approach to software telemetry, check out Jamie Riedesel’s Software Telemetry (Manning, 2021).

Logs are records of specific events. When something happens in our system--for example when we receive a request or when an error happens, we issue a log to keep a record of that event. Logs help us understand what is going on with our system, what endpoints are being used, what errors are being raised, and so on. However, as shown in figure 11.2, it’s quite difficult to connect the dots with logs. For instance, when an error is raised, it’s highly difficult to correlate that error to a specific request log. To do that, we need traces.

Figure 11.2 Every step in processing an API request may raise a log for our records. Without something to correlate the logs, it becomes difficult to connect them all and get the full picture.

Traces are records of the multiple processing steps taken to process a request throughout our system. As such, traces help us connect the dots between multiple logs by correlating the different logs produced during successive stages of a process through the same trace ID. For example, as illustrated in figure 11.3, when we process an HTTP request, we perform various tasks, such as performing database transactions, calling other services, and carrying out some other business logic tasks. Each of these tasks can raise their own logs, and traces allow us to connect them all so that we can reconstruct the sequence of steps that went into processing each request.

Figure 11.3 Traces help us connect all the logs that were raised within the same context, for example, during the processing of an HTTP request.

Traces are particularly helpful in understanding distributed systems as they give us end-to-end visibility of a request’s lifecycle across service boundaries. Let’s see what this means. As illustrated in figure 11.4, in a distributed system, events may go through several processing steps across different services. For example, when a user requests a quote from a home insurance website, their details may be processed by a quote service, which calculates the result by collaborating with the user service, the risk assessment service, and the policy management service. During this process, any of the collaborating services, such as the risk assessment service, may fail, resulting in the inability of our website to produce the insurance quote.

Figure 11.4 In distributed architectures, it can be difficult to find the root cause of an error if we have no way of correlating logs from different services.

As illustrated in figure 11.4, when the erroring service fails, it produces an error log. Taken in isolation, these inform us of what went wrong with the failing service, and this is valuable information to reproduce the error, debug the service, and fix it. In figure 11.4, the risk assessment service fails with an internal server error. Upon inspecting the logs, we find out that the error was caused by a bad payload containing a malformed postal code address, which causes an error when the service searches the area for environmental risk. The question is how did the malformed postal code make its way to the risk assessment service? Without a mechanism to correlate logs from all services, it’s difficult to answer this question. The best we can do is add a postal code validator to the risk assessment service and respond with a 422 status when we receive invalid postal codes, as illustrated in figure 11.5.

Figure 11.5 To avoid internal server errors in the risk assessment service, we strengthen its data validation layer and make it return 422 responses when it receives invalid data.

The fix applied to the risk assessment service in figure 11.5 helps to prevent future server errors when it receives malformed postal codes. However, it does not really address the root cause of the problem, which is why did it get a malformed postal code in the first place? Until we resolve that problem, users who insert bad postal codes will continue to be unable to receive insurance policy quotes.

Service-level logs, like those illustrated in figures 11.4 and 11.5, help us improve our services individually, but they don’t help us improve our system since they do not give us the full context in which the error occurred, and therefore do not allow us to trace the root cause of the error. To find and understand the bigger context of an error, we use traces. As illustrated in figure 11.6, traces are logs with additional information about the context in which they occur, such as the downstream event that triggered the event in which the log occurred.

Figure 11.6 Adding traces to our logs allows us to correlate them together and determine the original input that caused an error in an upstream service.

Traces give us the full picture of an error and help us determine its root cause. For example, the malformed postal code detail that triggered an error in the risk assessment service could have been caused by a mistake in the UI when capturing user details. Upon closer inspection, we could determine that the UI failed to make the postal code a mandatory field and instead sent over a default dummy value. The dummy value isn’t a valid postal code and therefore causes the risk assessment service to fail. As illustrated in figure 11.6, by using traces, we can obtain the full chain of events that led to this error and identify the root cause of the problem.

Traces foster cross-team collaboration by allowing us to identify the components that are involved in an error. When we do this exercise, we get to think about the overall improvements that are needed in our system to prevent such errors in the future, as opposed to improving individual services only.

Finally, metrics are measures of things that happen in our applications, and they capture details about resource usage and the behavior of our systems. As illustrated in figure 11.7, we have different kinds of metrics, such as low-level operating system statistics, and higher-level data like response latency and requests per second. Operating system metrics tells us about resource utilization, and it includes metrics such as CPU and memory usage, disk space, and so on. Those metrics tell us about the health of our system and whether we need to scale our resources.

Figure 11.7 Low-level performance metrics help us understand the state of our system, while higher-level API metrics help us understand how users interact with our APIs.

Higher-level metrics, such as the number of requests per second or average response latency, give us insight into how our users interact with our APIs, and they’re very helpful in detecting threats to our system. As illustrated in figure 11.8, keeping track of these metrics helps us identify outliers, like users sending unusual amounts of requests to our APIs or causing slower-than-usual response times. When we put this information together with logs and traces, we have right setup to track down and identify threats to our system and the malicious actors behind them.

Figure 11.8 By tracking higher-level metrics about API usage, we can detect outliers and potential threats. In this case, we observe that user cc35ff86 is sending a very unusual amount of requests and that could indicate that they are a malicious actor.

Now that we know what logs, traces, and metrics are, and how we leverage them, let’s see we set up our system to produce them. The critical concept here is instrumentation, and that’s the topic of the next section!

11.3 Instrumenting APIs

Telemetry is a fundamental feature of software applications. Telemetry informs us whether our applications are running well or have issues. However, just producing telemetry data isn’t very useful – we need to collect, store, and process that data in a way that allows us to draw insights from it. We call the practice of getting our applications ready to produce telemetry and collecting telemetry data instrumentation. Instrumentation makes our systems observable and allow us to detect anomalies and threats. For example, as shown in figure 11.9, if we want to understand how long it takes our application to run certain database queries, we need logs about those queries so that we can analyze them and obtain the relevant metrics. Changing our code to produce those logs is instrumentation.

Figure 11.9 Collecting logs about the processes that happen inside our applications, such as database queries, can be very helpful when debugging a problem or researching a cybersecurity incident. Such logs aren’t emitted by default, so we must make manual changes to our code to produce them.

Some instrumentation comes right out of the box. For example, most web frameworks produce request and response logs on every request that comes to our application. We simply need to capture those logs and store them in a way that makes it easy to draw insights from them. Some observability frameworks, like OpenTelemetry, also support auto-instrumentation. The idea behind auto-instrumentation is to get our applications ready to produce useful signals without having to make any code changes. We simply install a few dependencies, run a few commands to wire up our application with OpenTelemetry, and we are ready to start getting signals.

OpenTelemetry’s website maintains a list of zero-code instrumentation guides for the most popular software development languages, including Go, C#, PHP, Python, Java, and JavaScript (https://opentelemetry.io/docs/zero-code/). Let’s illustrate the auto-instrumentation process for Python with a practical example. As we’ve done in previous chapters, we use a FastAPI application as an example, and the uv dependency management tool to install our dependencies. Check ch07/README.md in this book’s GitHub repository (https://github.com/abunuwas/secure-apis/) for more details on installing Python, uv, and details about the dependencies.

First, let’s create a virtual environment for our project and activate the virtual environment:

uv init otel-fastapi && cd otel-fastapi && source .venv/bin/activate

Now let’s install FastAPI and Uvicorn:

uv add fastapi uvicorn

Create a file named server.py within the project’s folder and copy the code under ch11/server.py into that file. Server.py implements a simple API that we’ll use to illustrate the logging capabilities of OpenTelemetry’s auto-instrumentation packages. Next, we install the essential libraries we need to add zero-code instrumentation to our Python application:

uv add opentelemetry-distro opentelemetry-exporter-otlp

opentelemetry-distro brings all the utilities we need to start producing structured OpenTelemetry logs, and opentelemetry-exporter-otlp is a library that allows us to export those logs to a log collector of choice. The next step is to bring the relevant instrumentation libraries for our project packages, such as FastAPI and Uvicorn. We do it with the following command:

opentelemetry-bootstrap -a install

The above command reads our list of dependencies and install their corresponding instrumentation packages if available. To install the packages, opentelemetry-bootstrap uses the pip module. If pip not available in your environment and each package installation fails with a message along the lines of “No module named pip install”, run the following command to make pip available in your virtual environment:

python -m ensurepip

Then run the opentelemetry-bootstrap -a install again to install the instrumentation packages. Once you’ve done this, you can run your FastAPI application and start getting structured logs for free. Use the following command to instrument the application and export your logs to the console:

opentelemetry-instrument --traces_exporter console
[CA]--metrics_exporter console --logs_exporter console
[CA] uvicorn server:server

TIP.

Uvicorn’s --reload flag reloads the application when the code changes and it’s very useful when doing local development, but it interferes with the instrumentation libraries’ ability to hook into processes and export their outputs. If you want to test your OpenTelemetry’s instrumentation in your local server, don’t use Uvicorn’s --reload flag.

The opentelemetry-instrument command instructs the instrumentation to export all logs, traces, and metrics to the console. If you visit the API’s Swagger page at http://localhost:8000/docs and call any of the endpoints, like GET /books, you’ll see OpenTelemetry’s structured logs, in JSON format, in your terminal. You get three logs for every request that represent three different events: 1) the beginning of the HTTP response process; 2) preparation of the response body; 3) request details. The type of event is described under the attributes field, which in the first event is {"asgi.event.type": "http.response.start", "http.status_code": 200}, in the second event {"asgi.event.type": "http.response.body"}, and in the third event it’s a full list of details about the request, as shown in listing 11.1.

We call these events spans, and they represent steps in the processing of each HTTP request. You can see they are all part of the same process because they all share the same trace ID, which is available under the context.trace_id field. Listing 11.1 contains an example of span log with some lines truncated with ellipses to shorten the length of the listing.

Listing 11.1 Example of OpenTelemetry’s span log

{
    "name": "GET /books",
    "context": {
        "trace_id": "0x3b0367766ff238a8f6d7bdc8dc4b375b",
        "span_id": "0xe10f1dab2fc28c34",
        "trace_state": "[]"
    },
    "kind": "SpanKind.SERVER",
    "parent_id": null,
    "start_time": "2025-04-14T10:09:38.531298Z",
    "end_time": "2025-04-14T10:09:38.535245Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "http.scheme": "http",
        "http.host": "127.0.0.1:8000",
        "net.host.port": 8000,
        "http.flavor": "1.1",
        "http.target": "/books",
        "http.url": "http://127.0.0.1:8000/books",
        "http.method": "GET",
        "http.server_name": "localhost:8000",
        "http.user_agent": "Mozilla/5.0 [...],
        "net.peer.ip": "127.0.0.1",
        "net.peer.port": 50728,
        "http.route": "/books",
        "http.status_code": 200
    },
    [...]
    "resource": {
        "attributes": {
            [...]
        },
        "schema_url": ""
    }
}

As you can see in listing 11.1, the logs produced by OpenTelemetry’s auto-instrumentation packages contain a great deal of details about each request and how it was processed. If your server raises an exception while processing a request, OpenTelemetry’s auto-instrumentation will also capture that exception and produce a span log for it with details about the error raised, the error message, and the stack trace. With the help of the trace ID, you can pull all the associated spans in your logs to reconstruct the series of events that led to that error. Now that we know how to instrument our APIs and leverage OpenTelemetry’s logs, let’s see how we log custom events.

11.4 Logging custom events

Out-of-the-box signals produced by OpenTelemetry’s zero-code instrumentation libraries, like those we discussed in the previous section, help us understand generic behavior about our applications. They give us insights into the number of requests per second, request latencies, most commonly used endpoints, and so on. As illustrated in figure 11.10, these are signals about things that happen around our application. That’s very useful, but sometimes, we want to get a deeper understanding of our application by looking at signals raised from within our business layer. For example, we may want to know how long it takes to run calls against external service dependencies, certain database queries, or certain data manipulation and transformation tasks. As illustrated in figure 11.10, those signals come from inside of our application, and raising them requires manual changes to our code. Once we produce those logs, we simply need to capture and store them in a way that makes it easy to analyze them and draw insights.

Figure 11.10 Auto-instrumentation logs only capture events around the edges of our application, like arriving requests. To understand what happens inside our application, we must produce custom logs.

To output custom logs using OpenTelemetry’s format, we use a tracer object. Listing 11.2 shows how to generate a new span for a given trace within the POST /carts controller. Within the custom span, we capture details about the user and their order. As shown in listing 11.2, we obtain a tracer object by calling the get_tracer() function from Python’s opentelemetry library. get_tracer() has one required argument, which is the name of the module or the namespace within which we want to produce the traces. We use the tracer’s start_as_current_span() context manager method, which brings the context about the trace for the current request. This allows us to produce a span that is correlated with all other spans for the current request through the trace ID. Once we have the span object, we set the desired attributes within the span using the span object’s set_attribute() method, as illustrated in lines 13 and 14 of listing 11.2.

DEFINITION

In observability, cardinality refers to the number of unique combinations of attributes in a log event. It’s good practice to aim for high cardinality in our logs, which means their attributes make them highly unique. High cardinality makes our logs easier to find and context specific, which is helpful when debugging and tracing a problem.

Enriching our logs with custom event attributes makes them useful and informative when debugging a problem. For example, the code in listing 11.2 enriches the log with information about the user who added items to their cart using the access token’s sub claim (see section 7.2 for a refresher of token claims), and the full payload describing the items added to the cart. Information like this increases the cardinality of our logs, which makes our logs easier to find and more context specific.

Listing 11.2 Producing custom spans

# file: ch12/server.py
[...]
from opentelemetry import trace
[...]

@server.post("/carts", response_model=GetCart)
async def create_cart(
        cart_details: CreateCart,
        user_claims: UserClaims = Depends(authorize_access)
):
    cart = cart_details.dict()
    with tracer.start_as_current_span("cart.create") as span:
        span.set_attribute("user.sub", user_claims.sub)
        span.set_attribute("cart", cart_details.model_dump_json())
    [...]

To test the code in listing 11.2 run the server using the following command:

opentelemetry-instrument --traces_exporter console
[CA] --metrics_exporter none uvicorn server:server

Visit the API’s Swagger page under http://localhost:8000/docs and obtain an access token by sending a request to the POST /login endpoint, and use the token to authorize your requests as per the instructions in the ch11/README.md file of the book’s repository. Then, send a request to the POST /carts endpoint. In the terminal, you’ll now see 5 logs. Three of those logs represent the steps we discussed in section 11.3, namely the beginning of the HTTP response process (1), the preparation of the response body (2), and request details (3). In addition to those spans, auto-instrumentation generates one more log for POST requests with the following attribute: {"asgi.event.type": "http.request"}. The fifth log is our custom span, with the custom attributes we set in listing 11.2: user.sub and cart. Listing 11.3 illustrates an example of our custom span.

Listing 11.3 Details of a custom span

{
    "name": "cart.create",
    "context": {
        "trace_id": "0x3b32c85069830eabd4cbdc31e2179080",
        "span_id": "0x15d6be9f9da2a8a2",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0x260357a9703932fb",
    "start_time": "2025-04-15T10:26:24.194111Z",
    "end_time": "2025-04-15T10:26:24.202825Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "user.sub": "joe",
        "cart": "{\"books\":[{\"book_id\":1,\"quantity\":1}]}"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            [...]
        },
        "schema_url": ""
    }
}

As you can see in listing 11.3, the custom span contains a trace ID and a span ID. If you compare both values with the rest of the spans in the terminal, you’ll see that the span ID is unique, while the trace ID is shared with the four other logs that were raised for this request. If you produce more spans within the context of a request, all of them will share the same trace ID. This is incredibly powerful when debugging issues with your APIs and when investigating potential security exploits and abuses. In the remainder of this chapter, you’ll learn how to do just that.

11.5 Detecting input-based attacks

Observability is key to understanding the security posture of our systems, knowing when we are under attack, and reacting to threats. But what signals are we looking for from a security standpoint? What exactly do we need to log, how, and when? In previous sections, we’ve learned how to leverage instrumentation to produce signals such as logs, traces, and metrics. The question we address now is what signals we need to capture from a security point of view so that we can detect and analyze API security threats.

In this and the following sections, you learn what to look for in your application logs to detect various kinds of attacks on your APIs. We begin in this section by collecting evidence of input-based attacks.

As we learned in section 11.2, events are things like requests, errors, or any other relevant incident that happens in our system. From a security perspective, all these events are useful for detecting unauthorized access to our APIs, malicious input, and abuse of business logic and flows. If a request contains a malicious SQL or script injection, we want to keep a record of it. If the request resulted in an unauthorized (401 or 403 status codes) or a malformed data (400 or 422 status codes) response, we want to keep a record of it. Keep track of your response latencies–if a request takes a long time to process, it may indicate that the request included unusual and perhaps malicious input.

To make our logs useful for threat detection and analysis, we want to capture as many details as possible of each event. For example, when logging request details, it’s useful to capture every detail of the request, including its headers, request body, and full URL. Listing 11.4 illustrates what a complete request log may look like.

Listing 11.4 Logging full request details

{
    "name": "cart.create",
    "context": {
        "trace_id": "0x06210e62e32c09ae3bd66e3348dc38cf",
        "span_id": "0x53b11bde82bb6908",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xcdc3779f2a3d0a1a",
    "start_time": "2025-04-18T13:08:44.579768Z",
    "end_time": "2025-04-18T13:08:44.581234Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "user.sub": "string",
        "request.headers": "{\"host\": \"localhost:8000\",
[CA]\"connection\": \"keep-alive\", \"content-length\": \"72\",
[CA]\"sec-ch-ua-platform\": \"\\\"macOS\\\"\", \"authorization\": \"Bearer
[CA]eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzdHJpbmcifQ.CFahWaR1IaK
[CA]MgobCPsc9cMI3PxKuMT7vOyjTEXJ5g7s\", \"user-agent\": \"Mozilla/5.0
[CA](Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like
[CA]Gecko) Chrome/135.0.0.0 Safari/537.36\",
[CA]\"accept\": \"application/json\", \"sec-ch-ua\": \"\\\"Google
[CA]Chrome\\\";v=\\\"135\\\", \\\"Not-A.Brand\\\";v=\\\"8\\\",
[CA]\\\"Chromium\\\";v=\\\"135\\\"\",
[CA]\"content-type\": \"application/json\", \"sec-ch-ua-mobile\": \"?0\",
[CA]\"origin\": \"http://localhost:8000\", \"sec-fetch-site\": \"same-
[CA]origin\", \"sec-fetch-mode\": \"cors\", \"sec-fetch-dest\": \"empty\",
[CA]\"referer\": \"http://localhost:8000/docs\",
[CA]\"accept-encoding\": \"gzip, deflate, br, zstd\",
[CA]\"accept-language\": \"en-US,en-GB;q=0.9,en;q=0.8,es-
[CA]ES;q=0.7,es;q=0.6,ar-TN;q=0.5,ar;q=0.4,ko-KR;q=0.3,ko;q=0.2\",
[CA]\"cookie\": \"env=graphiql:disable\"}",
        "http.request_body": "{\"books\":[{\"book_id\":1,\"quantity\":1}]}"
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            [...]
        },
        "schema_url": ""
    }
}

Capturing full request details in your logs is useful for reproducing the exact behavior caused by a threat actor. This is very helpful for debugging and investigating security problems – you simply capture the request details and use them to send the same request to your API. You can do this with your API running in your local host, and with the help of breakpoints, analyze how exactly the request details affect the state of your system.

As you can see, capturing the full request details is incredibly helpful for investigating and reproducing security vulnerabilities in your APIs. However, if you’ve been in this industry for a minute, you know that logging everything isn’t always feasible, or even desirable. Request data often contains plenty of sensitive information, especially in sectors like healthcare, insurance, and banking. You can’t just store sensitive data in your logs and assume it will be fine. Every piece of data you own is a liability for your organization, and if you splash your logs with personally identifiable information (PII) and other sensitive data, you’ll put your logs in scope of regulatory frameworks like GDPR, PCI-DSS, HIPAA, and others. This means your logs will need very strict access controls, backup, and additional processes that will make them unusable for research and monitoring purposes. Besides, you’ll be setting yourself up for a data breach.

To avoid leaking sensitive data in your logs, identify which fields in your API schemas are likely to contain sensitive data, and mask those fields or remove them from the payloads before logging them. For example, in listing 11.5, we log the details of requests sent to the POST /orders endpoint. The payload for such request contains credit card information in the credit_card field, so we exclude that field from the logs when we dump the payload into the span’s attributes.

Listing 11.5 Excluding sensitive fields from the logs

# file: ch12/server.py
[...]
@server.post(
    "/orders",
    response_model=GetOrder,
    status_code=status.HTTP_201_CREATED
)
async def place_order(
        request: Request,
        order_details: PlaceOrder,
        user_claims: UserClaims = Depends(authorize_access)
):
    with tracer.start_as_current_span("cart.create") as span:
        span.set_attribute("user.sub", user_claims.sub)
        span.set_attribute(
            "request.headers", json.dumps(dict(request.headers))
        )
        span.set_attribute(
            "http.request_body",
            order_details.model_dump_json(exclude=["credit_card"])
        )
    [...]

Masking sensitive fields poses a problem: what if the attack is carried out through one of those sensitive fields? How do we know if the request contained an attack? There’s no fail-safe way to capture this information if we want to keep sensitive hidden. Sophisticated threat detection tools capable of identifying malicious injection attacks, such as web application firewalls (see section 9.4 for a discussion of this topic), use complex rules and often leverage machine learning models to identify such threats. If you can get a signal from your WAF that a request contains a malicious injection, see if you can configure it to capture the log and annotate it with a field that clarifies whether the request contains an injection attack.

A common concern that comes up when discussing leveraging logs for security is their impact on performance. Many architects and technical leaders worry that liberally producing a large amount of logs may take a hit on performance, making their APIs busier and slower. In my experience, this is especially true when logging to files. If that’s your case, consider using a more modern logging pattern, such as writing your logs to the standard output (stdout). As illustrated in figure 11.11, the idea is to treat your logs as streams and use a log router like Logplex or Fluentd to collect the logs and send them into a log aggregator, where you can store and analyze them.

DEFINITION

Sidecar is a pattern commonly used in container applications to run secondary processes with the main application. A sidecar container is a secondary container that runs alongside the primary application container. Sidecar containers provide supporting functionality such as logging, configuration management, and others. In container orchestration frameworks like Kuberentes, a sidecar container runs together with the primary application container within the same pod.

The logging pattern shown in figure 11.11 works especially well with container-based deployments, where you can easily run a log collector as an agent in a sidecar container, but it can be adapted to any kind of architecture and deployment model. Many threat detection and management vendors, such as Noname and Traceable, use the agent model to capture application logs and analyze them, and there’s no reason why you shouldn’t be able to implement the same pattern.

Figure 11.11 A common pattern for log collection and management is to run a logs collector agent together with the application. In container-based deployments, the agent runs as a sidecar container.

Request logs are very helpful to help us investigate security attacks against our APIs, and they are particularly useful for detecting attacks such as malicious injections, which are easily identifiable in user input field. However, logs alone won’t help us identify less obvious attacks that are executed over a number of requests. In the next section, we turn to the requirements for tackling those kinds of attacks.

11.6 Detecting endpoint abuse attacks

In January 2021, cybersecurity researcher Jan Masters discovered a flaw in the API of Peloton, a popular exercise equipment manufacturer. Masters discovered a few endpoints from Peloton’s REST API and a few operations from its GraphQL API that failed to enforce user credentials and allowed unauthorized access to other users’ personal data. Thankfully, Masters disclosed the vulnerability responsibly, which allowed Peloton to fix it before it became a known exploit (https://www.pentestpartners.com/security-blog/tour-de-peloton-exposed-user-data/).

Peloton’s security incident poses an interesting question: is there anything we can do to detect unauthorized access to our APIs before a security researcher, or worse, a threat actor, discovers the flaw? We certainly can. The first step is to understand how this type of vulnerability will be exploited. In Peloton’s case, the exploit involved two steps:

Find existing users with the GET /api/user/search/{username} endpoint. This endpoint allowed you to find other users by username. The response included a few personal user details, including their user ID.

Pulling full user profiles using the POST /stats/workouts/details endpoint. This endpoint required a request body containing the user IDs whose stats you want to see, and before the vulnerability was fixed, it would return full user profiles with personal details, including the user’s age, location, gender, and so on.

As illustrated in figure 11.12, threat actors would exploit the vulnerability by calling the GET /api/user/search/{username} endpoint repeatedly to pull user identifiers, then feed those user IDs to the POST /stats/workout/details endpoint to retrieve their personal details.

Figure 11.12 Peloton’s user search endpoint could be exploited to enumerate existing users and pull their details. The signature of this type of abuse is the repeated number of calls against the same endpoint to scrape the data.

From an endpoint abuse perspective, the relevant step is the call to the GET /api/user/search/{username} endpoint, which happens repeatedly to pull a list of existing users. We call the attack shown in figure 11.12 object enumeration–threat actors use the user search endpoint to get a list of existing users. The distinct signature of this attack is the number of repeated calls against the same endpoint. Contrary to the types of attacks we analyzed in section 11.5, the attack illustrated in figure 11.12 does not bear a clear footprint in the logs and therefore cannot be detected from a single log event. In this case, we are looking for an interaction pattern against our endpoints. Let’s see how we go about collecting this information, and importantly, how we can make such information actionable.

Since the signature of the attack in figure 11.12 is the repeated number of calls against a given endpoint, we want to track endpoint-level request metrics. Listing 11.6 illustrates how we capture such metrics using OpenTelemetry.

Listing 11.6 Capturing metrics with OpenTelemetry instrumentation

# file: ch12/server.py
[...]
from opentelemetry.metrics import get_meter
[...]
meter = get_meter("orders_meter")
list_books_counter = meter.create_counter("list_books")

@server.get("/books", response_model=ListBooks)
async def list_books():
    list_books_counter.add(1)
        [...]

Run the server again and send a few requests to the GET /books endpoint. Metrics logs show up every 60 seconds, so wait for that time the custom metric in your terminal. Listing 11.7 shows a sample of metrics log. Metrics logs contain a collection of default metrics provided by OpenTelemetry’s auto-instrumentation, such as the number of active requests, reponse latency, and response size. In this case, those metrics are scoped under the opentelemetry.instrumentation.fastapi namespace. Our custom metrics appear under the orders_meter scope, and in this example it includes a metric named list_books which counts the number of requests to the GET /books endpoint, and whose value in this example is 2.

Listing 11.7 Example of OpenTelemetry metrics log

{
  "resource_metrics": [
    {
      "resource": {
        "attributes": {
          [...]
        },
        "schema_url": ""
      },
      "scope_metrics": [
        {
          "scope": {
            "name": "opentelemetry.instrumentation.fastapi",
            [...]
          },
        }
        {
          "scope": {
               "name": "orders_meter",
               [...]
          },
          "metrics": [
            {
              "name": "list_books",
              "description": "",
              "unit": "",
              "data": {
              "data_points": [
              {
                "attributes": {},
                "start_time_unix_nano": 1745240652666090000,
                "time_unix_nano": 1745241786748827000,
                "value": 2,
                "exemplars": []
              }
            ],
          "aggregation_temporality": 2,
          "is_monotonic": true
        }
      }
    ],
  }
  [...]

Capturing endpoint-level call metrics will alert us if there’s an unusual spike in the number of calls against a certain endpoint. Keep track of the average number of requests per second and per minute, and create alerts for when requests go above the average. When an alert triggers, don’t take action immediately–investigate the issue to determine if the spike in requests is truly anomalous. If something is off and you are reasonably concerned, a good measure is to start rate limiting all requests to the endpoint under attack.

Tracking the number of requests per second helps us detect unusual activity, but it still leaves us in the dark about the origin of the offending requests, and therefore the countermeasure–rate limiting all requests to the endpoint–is vague and impacts the experience of all users. For a more effective response, we need to know if a specific threat actor is driving the spike, or whether the requests have a common origin, for example, if they come from a particular geolocation. If we have this knowledge, we can take measures to block certain actors from our website or rate limit requests coming from a specific region.

To track the origin of the request, include information about the client in your logs. Include the client’s IP and the device’s fingerprint, and if the endpoint is authenticated, include the user ID too. Listing 11.8 illustrates how we capture such details for protected endpoint such as POST /orders.

Listing 11.8 Capturing metrics with custom details

# file: ch12/server.py
[...]
meter = get_meter("orders_meter")
list_books_counter = meter.create_counter("list_books")
place_order_counter = meter.create_counter("place_order")

@server.post(
    "/orders", response_model=GetOrder,
    status_code=status.HTTP_201_CREATED,
)
async def place_order(
        request: Request,
        order_details: PlaceOrder,
        user_claims: UserClaims = Depends(authorize_access)
):
    place_order_counter.add(
        1, {
            "sub": user_claims.sub,
            "headers": json.dumps(dict(request.headers))
        }
    )
    [...]

Once you start capturing data about the origin of the request, run continuous analysis to identify unusual behavior from certain clients or users. During the analysis, resolve the geolocation of every IP to be able to capture trends by region. Before you take any action, you need to understand your data, so spend some time analyzing it and calculating what a reasonable number of requests per user, IP, device fingerprint, or region, for a given endpoint is. Once a certain user, client, or region goes above a reasonable threshold, take steps to rate limit them, and a certain user or client keeps reoffending, you may want to take steps to blacklist them from your website.

The measures discussed so far go a long way to protect our APIs from many common types of attacks, but there’s still one category of threats that they cannot fend off, that is attacks designed to abuse our business or user flows. In the next section, you learn to tackle such attacks.

11.7 Summary

API observability is the practice of collecting and continuously analyzing data about our APIs. This is crucial to understand how users engage with our APIs and to identify threats.

To enable observability, we collect telemetry data such as logs, traces, and metrics from our APIs:

Logs are records of specific events, they tell us that a request was sent to our servers or an error occurred.

Traces help us put various logs together when they belong in the same context, for example when they represent various steps in the processing of an HTTP request.

Metrics capture aggregate system behavior, such as the number of requests per second or average response times, and therefore help us identify unusual trends or patterns.

OpenTelemetry is a standard for generating, collecting, and analyzing telemetry data. OpenTelemetry provides libraries to auto-instrument our APIs to start producing telemetry data and enable observability.

Manual instrumentation is the practice of making changes to our code to produce custom telemetry data. This allows us to tailor our logs for security observability and helps us identify threats.

To identify input-based attacks, log all user input, ensuring you remove all sensitive data from the logs.

To identify endpoint abuse attacks, capture metrics about API usage to understand how users interact with your APIs and identify outliers.

12 Testing API security

This chapter covers

Creating a testing strategy tailored to our threat models

Testing API specifications to discover security by design flaws

Leveraging contract testing and fuzzing to ensure our APIs work as intended

Creating unit tests that help us assess the security posture of our APIs

Creating complex tests to identify vulnerabilities in our business logic and flows

As we build our APIs, the inevitable question is whether we are making them secure. If you’ve followed all the best practices described in this book, you’ve threat modeled your API design choices, considered tradeoffs, accepted some risks, and produced a sound implementation. The thing is, there’s only so much we can do using threat modeling techniques and following best practices. At some point, we need some tests that give us a good level of confidence that our APIs are working the way they’re supposed to, and that they’re not vulnerable to exploits.

The two most common approaches to assessing the security of API implementations are using a penetration testing service and automated API security testing tools. Both choices are good and have their own place in a security testing strategy.

Penetration testing services deliver high-quality results tailored to your APIs, especially when they’re conducted by experts in API security. However, such tests are expensive, slow, take days to complete, and don’t scale, since they require human experts looking at your API and understanding how it works. If you can afford it, I recommend you run a full penetration test against your APIs once a year, or over slightly longer intervals.

Automated API security testing tools, on the other hand, are fast. The quality of the results depends on the quality of the tool and how much high-quality configuration data you feed into it. Generally, automated security testing tools fail to develop an understanding of how your API works and hence are likely to miss business logic flaws, but they’re helpful to automate simple tests such as malicious injections, configuration checks, and so on..

Regardless of whether you use penetration testing services or automated testing tools, you can get a lot of benefit by laying out a custom security testing strategy for your APIs. You’ve designed, threat modelled, architected, and implemented your APIs, so you know everything there’s to know about them, and hence you’re in a unique position to design and put together security tests that target very specific flaws that only you know about. In this chapter, we’re going to leverage this knowledge to design an API security testing strategy, and we’ll see how we can put together simple tests that deliver tons of value for our API security posture.

12.1 Designing an API security testing strategy

So we want to put together some tests to assess the security of our API implementation. But where do we start? If you’ve read all the chapters in this book, you know there’re many different types of attacks that can be performed against your API, and many kinds of flaws that can be exploited for various purposes. Anything from SQL injection, access token tampering, algorithm confusion attacks, object enumeration, sensitive data disclosure, mass assignment, and pagination attacks, to more complex strategies such as business-flow-based exploits and abuses. With so many possibilities, how do we decide the best way to get started? In other words, what is the right security testing strategy for our APIs?

To design a solid security testing strategy for our APIs, we must tap into our knowledge of how the API works. A deep understanding of our API’s design, implementation, and threat models will help us put together the right security tests. For example, if we know that our API does not use structured access tokens such as JWTs to authenticate requests, and instead uses opaque tokens, then we can rule out an entire category of tests around token tampering, algorithm confusion attacks, and more. The more we leverage our knowledge of our APIs, the more focused and valuable our tests become.

Let’s put together a structured method for designing our security testing strategy. We begin with our threat model. Some threats are relatively easy to test and others require complex settings. For example, a broken authentication test may be as simple as sending a request without credentials to each of our endpoints, while testing sensitive business flows may require performing a series of complex operations during the test and running an assertion about the specific state of a resource. If we don’t have any security tests yet in place, our goal is to identify the low-hanging fruits first—that is, tests that require minimal setup, yet deliver high impact due to their security implications. Once we’ve written the simplest and most impactful tests, we can move on to more complex and less impactful ones.

To illustrate the process of coming up with a testing strategy, let’s work through a practical example. Imagine we run an online learning platform. The platform offers online self-paced courses created by independent instructors. The course content is valuable, and to access it, students must create an account and sign up for the course they want to take. When a student creates their account, they receive a one-time 80% discount voucher on any course. Our platform holds sensitive data about students, such as their interests, progress, commitment, and other factors that allow us to tailor the learning experience for them.

After various sessions of threat modeling, we have identified three major threats we want to prevent:

Unauthorized access to course content

Abuse of coupon codes

Unauthorized access to personal data via the admin portal

Let’s break down each threat model. The first threat is unauthorized access to course content, which can happen when we fail to enforce credentials. As illustrated in figure 12.1, to address this threat we must check whether users are authenticated and have valid credentials. Our website uses JWTs for authentication, so we must ensure we are not vulnerable to common JWT exploits such as tampering and algorithm confusion attacks.

Figure 12.1 Our threat model considers the possibility that unauthenticated users can access protected content, hence we must write tests that verify that our authentication layer works properly.

The second threat involves abusing our coupon codes. As stated earlier, every student gets a one-time 80% discount voucher when they create an account on our website. As illustrated in figure 12.2, the threat model here is failure to verify that the student claiming a voucher has legitimate access to it. This exploit means rogue users could create tons of new accounts to release new vouchers, and use them to pay for new courses with their main account. We want to create tests that check whether users can apply vouchers that don’t belong to them. This is also a great scenario to apply the threat detection strategies discussed in section 11.6.

Figure 12.2 Our threat model considers the possibility of users creating rogue accounts to use their discount code vouchers, effectively obtaining permanent discounts and compromising our business model.

Finally, our last threat model describes a privilege escalation attack that allows unauthorized users to access sensitive student data through the admin portal. As illustrated in figure 12.3, privilege escalation may occur if we fail to validate role-based access controls correctly. To see if our API is resilient to these threats, we want to write some tests that check if non-admin roles can access admin endpoints and functionality.

Figure 12.3 Our threat model has a scenario for privilege escalation where less privileged users, such as students, can access admin endpoints such as GET /admin/students. We must write tests to ensure this doesn’t happen.

Now that we know how to lay out a security testing strategy for our APIs, let’s get testing, beginning with the most basic type of test: testing the specification!

12.2 Discovering design security flaws in our APIs

Good security begins with a solid design. As we stated in section 3.1, our API design is a fundamental artifact in our security process, since it contains an exact description of how our API works, its constraints, and the expected attack surface. It’s best practice to consolidate our API design in a formal specification, since such documents can be formally analyzed and used for testing. In this section, you learn to leverage API specifications for design security testing. Whether you follow a design-first or code-first approach, the testing technique described in this section will help you gain visibility of design flaws in your API and therefore put you on a path towards better security.

The most popular tool for testing our API specifications for security flaws is Spectral (https://github.com/stoplightio/spectral). Spectral is an API linter; that is, a tool that checks that your specifications are correctly written. It’s very popular in API governance, and many teams around the world use it to enforce a common set of guidelines across their API documentation. Spectral’s basic linting capabilities can be enhanced with plugins, and one such plugin is an OWASP ruleset that analyzes API specifications looking for security flaws. It’s a very powerful plugin that brings tons of useful suggestions to improve our APIs. Let’s see how it works.

We’ll begin by installing Spectral and the OWASP plugin. Since Spectral is an npm package, we can install it with both npm and yarn. You’ll need a node.js runtime to run Spectral in your machine. If you don’t have node.js and npm, you can install them from the official website: https://nodejs.org/en/download. I highly recommend you install nvm too to help you manage node.js versions easily. You can download nvm from the same website. cd into folder ch12 from the book’s code repository, and run the following command to install Spectral:

npm install @stoplight/spectral

Then install the OWASP plugin with the following command:

npm install @stoplight/spectral-owasp-ruleset@^2.0

Finally, create a local configuration ruleset for Spectral that uses the OWASP plugin:

echo 'extends: ["@stoplight/spectral-owasp-ruleset"]' > .spectral.yaml

We are ready to run Spectral against our API specification. You can grab the specification from the book’s repository under ch12/openapi.json, or you can fetch it directly from your local server by starting it up and visiting http://localhost:8000/openapi.json. Please refer to ch12/README.md in the book’s code repository for further instructions on setting up your environment to run the local server. To run Spectral against the specification, use the following command:

npx spectral lint openapi.json --ruleset `pwd`/.spectral.yaml

If you run the above command against the specification, you’ll see that it raises 160 problems, including 75 errors and 85 warnings. This is a small set of errors, mostly because the API specification is generated from code and FastAPI happens to produce excellent specifications. However, the specification is not without faults, and Spectral is helpfully bringing those to our attention. Below is a sample of the output produced by Spectral:

9:14 error  owasp:api2:2023-write-restricted This write operation is not
[CA] protected by any security scheme. paths./students/register.post
[...]
714:17 warning  owasp:api4:2023-string-restricted Schema of type string
[CA] should specify a format, pattern, enum, or
[CA] const.components.schemas.ValidationError.properties.msg
718:18 error  owasp:api4:2023-string-limit Schema of type string must
[CA] specify maxLength, enum, or
[CA] const. components.schemas.ValidationError.properties.type
718:18 warning  owasp:api4:2023-string-restricted Schema of type string
[CA] should specify a format, pattern, enum, or
[CA] const. components.schemas.ValidationError.properties.type

✖ 160 problems (75 errors, 85 warnings, 0 infos, 0 hints)

Let’s break down all this analysis so we can make sense of it. We can classify Spectral’s feedback into three categories:

Authentication and access controls

Input constraints

API usability

With regards to authentication and access controls, Spectral reports that some of our endpoints are not authenticated. For example, the below message tells us that the POST /students/register endpoint is unprotected:

9:14 error  owasp:api2:2023-write-restricted This write operation is not
[CA] protected by any security scheme. paths./students/register.post

In most cases, endpoints are not authenticated by design, but a careful review of these detections may reveal unauthenticated endpoints that should protected. Spectral also tries to identify admin endpoints and analyzes their security requirements. For example, in our specification, it correctly identifies the GET /admin/students endpoint as an admin endpoint through the presence of the admin keyword in the URL path, as illustrated in the below message:

294:11 error  owasp:api5:2023-admin-security-unique Admin endpoint
[CA] /admin/students has the same security requirement as a non-admin
[CA] endpoint.  paths./admin/students.get.security[0]

Spectral highlights that this endpoint has the same security requirements as non-admin endpoints. While this could also be by design, it’s worth reviewing such detections and assess their security implications.

As for input constraints, Spectral raises detections for strings, integers, and arrays. The detections apply to both read-only and input schemas alike. For example, the following message says that the email field of the GetStudent schema does not have constraints such as maximum length, enumeration, or constant values:

545:19 error  owasp:api4:2023-string-limit Schema of type string must
[CA] specify maxLength, enum, or
[CA] const. components.schemas.GetStudent.properties.email

GetStudent is a read-only schema that we use to return student details in endpoints like POST /students/register and GET /admin/students. For read-only schemas such as GetStudent, lack of constraints is less of an issue, since we control how such models are rendered on the server. However, for input schemas, these detections are very much relevant. For example, the RegisterStudent schema, which is used in the POST /students/register endpoint to register a new student, lacks constraints across all its fields, including email, address, first_name, etc. The below message, for example, says that the last_name field in the RegisterStudent schema is unconstrained:

668:23 error  owasp:api4:2023-string-limit Schema of type string must
[CA] specify maxLength, enum, or
[CA] const. components.schemas.RegisterStudent.properties.last_name

As we saw in section 6.3, it’s good practice to constrain every field in our specification to prevent threat actors from abusing them by sending malicious payloads such as injection attacks, large payloads, or malicious regular expressions. However, it's not always possible to constrain a field. For example, in RegisterStudent, it’s not feasible to constrain the values of fields like first_name and last_name. As we discussed in chapter 11, our security by design process can only go so far, and it must be paired with a strong observability framework to detect and prevent threats in real time.

We also have useful detections for integer fields. For example, the below message says that the first parameter of the GET /courses endpoint doesn’t specify minimum and maximum boundaries:

93:22    error  owasp:api4:2023-integer-limit Schema of type integer must
[CA] specify minimum and maximum. paths./courses.get.parameters[0].schema

If you check the OpenAPI specification for this API, you’ll see that the first parameter in the parameters list of the GET /courses endpoint is page, which is a pagination parameter. The same detection applies to the second parameter, which is per_page. Lack of constraints in these fields may allow threat actors to run pagination attacks against our API, overwhelming our servers by asking for millions of items in a single request. It’s important that we review these kinds of detections and assess whether we can impose constraints that protect our services from such abuses.

If you go through Spectral’s output, you’ll also see detections for unconstrained arrays. For example, the below message says that the field lessons of the GetCourse schema doesn’t specify the maximum number of items in the array:

447:21 error  owasp:api4:2023-array-limit Schema of type array must specify
[CA] maxItems. components.schemas.GetCourse.properties.lessons

In this case, the unbounded array belongs in a read-only schema, which is generally less of a problem. However, even in cases like this, it could be worrying if a course ends up with a very large number of lessons and we return them all in one go -that could put unnecessary pressure on our server and compromise its performance and even its availability. A malicious actor could certainly take advantage of this feature to create a course with many lessons and introduce performance problems in our API. As always, use your judgement as to when and where it makes sense to paginate any collection of items.

In addition to the detections discussed so far, Spectral also recommends documenting error responses such as 401 (Unauthorized), 403 (Forbidden), and 500 (Internal server error), which are absent from our specification. While not strictly necessary from a security perspective, documenting error responses makes our APIs easier to consume by third parties. Another Spectral recommendation is to document rate-limiting headers. As we learned in section 5.1, rate limiting headers tell our consumers how often they can hit our endpoints. This is especially important if the API is for external consumers, so that they can consume our APIs responsibly and so that they know when and why they are being rate-limited.

160 is relatively small number of detections for a single API specification. In my consulting work helping organizations secure their APIs, I routinely came across specifications raising several hundreds and even thousands of detections. The goal with this exercise isn’t to address all the detections, but to use them as a baseline for analysis of where you stand from a security by design perspective, get visibility and raise awareness of the potential security problems that your API may encounter, and decide how you want to get ready to tackle those scenarios.

Now that we know how to test and ensure that our API design is secure, the next step is verifying that our implementation conforms to that design. It’s no use going through the effort of constraining all input parameters in our API specification if the server does not implement such constraints, and sadly, differences between our implementation and the specification are very common. In the next section, you learn how to improve the server implementation using the API specification as a validation tool. Let’s see how that works!

12.3 Leveraging fuzzing and contract testing

One of the most common sources of problems in API security is the difference between the server implementation and the API specification. For example, as illustrated in figure 12.4, the API specification may say that a certain endpoint exists, but that endpoint cannot be found in our server. The reverse is also true, and more concerning: our server may expose endpoints that are not documented in the specification, also known as shadow endpoints (see section 3.5). This difference between the API specification and the server implementation is known as API drift.

Figure 12.4 API drift is the difference between the API specification and the implementation. In this example, the specification documents one endpoint, GET /courses/{course_id}/details, that is not implemented, and hence it responds with 404 responses when users request that endpoint.

API drift affects every element of an API, including endpoints, parameters, security schemes, etc. For example, as illustrated in figure 12.5, the API specification may say that the endpoint GET /courses supports a query parameter named topic that allows us to filter courses by topic. However, when we try to use that parameter, we may discover that it makes no difference in the results, i.e., the API doesn’t make use of it, or we may get an error response saying that such parameter isn’t supported.

Figure 12.5 API drift also happens with query parameters, such as the parameter topic in this example, which is documented in the specification, but is not implemented and hence the API responds with a 422 status code when clients use it.

Equally, the API specification may say that the GET /courses endpoint supports pagination parameters named page and per_page, as illustrated in figure 12.6. The per_page parameter allows us to specify how many items we want to retrieve per page, and according to the specification, it has no boundaries, i.e., we can request as many items as we want. However, when we ask the API to give us 100 items per page, it returns only 25. There’s clearly a drift between the implementation and the specification, with the implementation enforcing an undocumented maximum limit of items per page.

Figure 12.6 API drift also happens when the implementation behaves differently from the specification. In this example, the specification says the per_page parameter has no boundaries, but the implementation has a limit of 25 items per page, resulting in frustrated users who want to see more items per page.

API drift is a source of frustration for API consumers. If you are a provider of services like payments, banking, risk assessments, pension management, etc., and you offer those services over APIs, your customers rely on your API specifications to understand how to integrate with your services. If the specifications don’t match the server implementation, the result is poor developer experience (DevEx), unexpected integration errors, and unhappy customers ready to leave your services for a competitor.

And it’s not just a DevEx problem, it’s also a security problem. Undocumented API endpoints and parameters often go untested and may expose unintended behaviors, and API drift means some of the constraints documented to our input parameters may not be implemented by the server. For example, the reverse of the example in figure 12.6 happens if the specification documents a maximum value of 25 for the per_page parameter but the implementation fails to apply such limit, effectively allowing threat actors to overwhelm our services by requesting millions of items in a single request.

To prevent API drift, we test the server implementation against the API specification. This is also known as contract testing. The idea is that the API specification represents a contract between the API server and its consumers. We are making sure that the API behaves as intended, or in other words, that it abides by the contract (aka the specification). Contract testing is especially useful when the API specification is written or modified by hand, but it also raises important observations when the specification is automatically generated from code. The API specification we use in this chapter is generated automatically from code, and as you’ll see, contract testing will help us catch a few implementation issues and potential security exploits.

There are various tools available for contract testing, with the most popular being schemathesis (https://github.com/schemathesis/schemathesis), Microcks (https://github.com/microcks/microcks), Dredd (https://github.com/apiaryio/dredd), Restler-fuzzer (https://github.com/microsoft/restler-fuzzer), and API-fuzzer (https://github.com/Fuzzapi/API-fuzzer). In this section, we use schemathesis to illustrate contract testing due to its simplicity of use, but I encourage you to check out all the tools, especially Microcks, which provides contract testing capabilities for API consumers too.

Schemathesis works by analyzing the API specification and dynamically creating tests cases that will check if the server implementation conforms with the specification. The approach schemathesis uses to do this is known as fuzzing. Fuzzing works by sending malformed input to the server to evaluate how it reacts. In many cases, APIs crash with errors or behave in unexpected ways when subjected to a fuzzing test, and this analysis sheds light on how threat actors could abuse our APIs and what we need to do to prevent such abuses.

Schemathesis is a Python package, hence we can install it within our virtual environment with uv using the following command:

uv add --dev schemathesis

If your project isn’t in Python and/or you don’t use uv, you can install schemathesis with pip using the following command:

pip install schemathesis

Schemathesis runs against a live server, so before we run our tests, let’s get the server up and running with the following command:

uvicorn server:server --reload

Additionally, let’s obtain an access token. Some of the endpoints are authenticated, and schemathesis won’t be able to raise good detections if it can’t get passed our authorization layer. To obtain an authentication token, visit the API’s Swagger page under http://localhost:8000/docs and use the /students/register endpoint to register a new account, and then use the /students/login endpoint to obtain a token with the newly created account. With a fresh token at hand, we are ready to run schemathesis against our API. Keep the API server running in one terminal, and open another terminal to run schemathesis with the following command:

schemathesis run openapi.json --base-url http://localhost:8000
[CA] --experimental=openapi-3.1 -H 'Authorization: Bearer <token>'
[CA] --checks=all

We use schemathesis’s run command to run the test, followed by an argument that represents the path to our OpenAPI specification, which is openapi.json in this case, and the following flags:

--base-url: the base URL of our server, which is http://localhost:8000.

--experimental: since the API specification uses OpenAPI version 3.1 and that version, at the time of writing, is in experimental support in schemathesis, we add the --experimental flag with the value of openapi-3.1.

-H: we use the -H flag to instruct schemathesis to include certain headers in the requests during the test. In this case, we use it to authenticate the requests using the Authorization header, and we set its value to the access token we obtained in the previous paragraph, together with the Bearer prefix.

--checks=all: instructs schemathesis to run all validation tests. If we omit the --checks=all flag, schemathesis checks only if the server raises internal server errors.

You can follow schemathesis’s progress on the terminal from which you ran the schemathesis command, and you can see the requests it runs against your API on the terminal that runs your API server. As you can see, schemathesis runs dozens to hundreds of tests per endpoint, testing various combinations of malformed payloads, to discover how and when our server breaks or fails to behave correctly. In the current scan, five of the nine endpoints fail to pass the fuzzing test. Below is schemathesis’s test execution summary:

================= Schemathesis test session starts =================
Schema location: file:///secure_apis/ch12/openapi.json
Base URL: http://localhost:8000
Specification version: Open API 3.1.0
Random seed: 63443938763838035015416547015700196325
Workers: 1
Collected API operations: 9
Collected API links: 0
API probing: SUCCESS
Schema analysis: SKIP

POST /students/register F.                                           [ 11%]
POST /students/login .                                               [ 22%]
GET /courses .                                                       [ 33%]
GET /courses/{course_id} .                                           [ 44%]
GET /courses/{course_id}/lessons/{lesson_id} .                       [ 55%]
POST /courses/{course_id}/register F                                 [ 66%]
GET /admin/students F                                                [ 77%]
GET /me F                                                            [ 88%]
GET /registrations/{registration_id} F                               [100%]

And below is the test results summary:

======================== SUMMARY ========================
Performed checks:
    not_a_server_error                    504 / 508 passed          FAILED
    status_code_conformance               504 / 508 passed          FAILED
    content_type_conformance              508 / 508 passed          PASSED
    response_headers_conformance          508 / 508 passed          PASSED
    response_schema_conformance           506 / 508 passed          FAILED
    negative_data_rejection               508 / 508 passed          PASSED
    ignored_auth                          504 / 508 passed          FAILED

As you can see, schemathesis ran 508 tests per endpoint, and in some cases, some of the tests failed. When an endpoint fails, schemathesis tells us exactly how it failed and how to reproduce the result with a cURL command. Since schemathesis uses a randomizer strategy to generate test cases, the results you obtain may look different from mine. In my case, for example, the POST /students/register endpoint failed with the following request:

curl -X POST -H 'Content-Type: application/json' -d '{"address": "",
[CA] "date_birth": "2000-01-01T00:00:00Z", "email": "", "first_name": "",
[CA] "last_name": "", "password": "", "phone_number": "\u0000"}'
[CA] http://localhost:8000/students/register

If you run that request against your local server, you get a 500 server error back. Looking at the stack traces in the server’s terminal, we see the error reproduced in listing 12.1. The important line in the stack trace is where it says PostgreSQL text fields cannot contain NUL (0x00) bytes. If you look at the request payload above, it’s not clear which field contains a NUL (0x00) value, but a closer look at the stack trace gives us more details. In the last few lines of the trace, where we see the query’s parameter values, we have the following mapping: 'phone_number': '\x00'. The NUL value is in the phone number. In the payload, this value is represented as \u0000. Both \u0000 and \x00 are representations of the NUL character, \u0000 uses JSON Unicode’s escape notation while \x00 uses hexadecimal’s escape notation. The test generated by schemathesis is successfully disguising a null value as a string, hence bypassing our data validation layer. This type of attack is known as null byte injection, and in some cases, it can be exploited to inject malicious code into our server. For more details on null byte injection, check out CWE-158 (https://cwe.mitre.org/data/definitions/158.html).

Listing 12.1 Stack trace error from schemathesis test

[...]
sqlalchemy.exc.DataError: (psycopg.DataError) PostgreSQL text fields cannot
[CA] contain NUL (0x00) bytes
[SQL: INSERT INTO student (email, password, first_name, last_name,
[CA] date_birth, phone_number, address, id, created) VALUES
[CA] (%(email)s::VARCHAR, %(password)s::VARCHAR, %(first_name)s::VARCHAR,
[CA] %(last_name)s::VARCHAR, %(date_birth)s::TIMESTAMP WITHOUT TIME ZONE,
[CA] %(phone_number)s::VARCHAR, %(address)s::VARCHAR, %(id)s::UUID,
[CA] %(created)s::TIMESTAMP WITH TIME ZONE)]
[parameters: {'email': '', 'password': '', 'first_name': '', 'last_name':
[CA] '', 'date_birth': datetime.datetime(2000, 1, 1, 0, 0,
[CA] tzinfo=TzInfo(UTC)), 'phone_number': '\x00', 'address': '', 'id':
[CA] UUID('3cfbeb71-b87c-4368-8bd0-7b7cd62ba8ce'), 'created':
[CA] datetime.datetime(2025, 5, 7, 10, 7, 13, 509445,
[CA] tzinfo=datetime.timezone.utc)}]

To address the null byte injection attack illustrated in listing 12.1, you can enhance your data validation models to reject payloads when a field contains the \x00 string. If you go through the rest of the error reports raised by schemathesis, you’ll be in a strong position to prevent API abuses that take advantage of flawed data validation models. To learn more about fuzzing, contract testing, and schemathesis, check out chapter 12 of my book Microservice APIs (Manning, 2022).

Using standard API testing tools like we’ve seen here and in section 12.2 is very useful and it goes a long way to harden our APIs, but such tools only scratch the surface of API security testing. To uncover deeper issues in our implementation, we have to roll up our sleeves and write down some unit tests that reach the API’s business layer and reproduce our threat models. In the next section, we begin this effort with low-hanging fruits that yield the most value.

12.4 Automating access control tests

Most APIs use authentication to control access to protected resources. For example, a healthcare API that manages appointments between patients and doctors will require patients to create an account and will store patient data with bindings to their respective accounts. When a patient tries to access data about their appointments, they must be authenticated, and we want to make sure the API only returns data that belongs to the patient. Furthermore, the API may have a few admin endpoints that only a few members of the staff can use to access data from all patients. All these are examples of access controls, and we want to make sure such access controls are correctly implemented to prevent unauthorized access and data leaks. Fortunately, these are some of the easiest tests we can write, so we’ll begin by writing unit tests for access control checks.

We can divide access control tests into four major groups:

Broken authentication tests: we check if the API enforces authentication credentials correctly. For example, we check if we can send an unauthenticated request to an authenticated endpoint. We can also test with malformed, expired, or tampered access tokens, etc.

Broken object-level authorization tests: we test if a user can access get unauthorized access to data from another user. For example, in a healthcare application, we check if patient A can access data from patient B.

Role-based access control tests: we test whether users can access data and functionality to their user role. For example, we can write a privilege escalation test to see if a normal user can access an admin endpoint.

Attribute-based access control tests: sometimes, access to certain data and functionality requires users to be in a specific state of have certain attributes. For example, a user may only be able to make payments through a payments application when their bank account has been verified, and we want to test if that condition is correctly enforced.

Let’s begin by creating a basic broken authentication test. The API has nine endpoints, of which five are or should be authenticated:

GET /courses/{course_id}/lessons/{lesson_id}

POST /courses/{course_id}/register

GET /admin/students

GET /me

GET /registrations/{registration_id}

The task here is to write a broken authentication test for each of these endpoints. For illustration purposes, we’ll create a test for the GET /courses/{course_id}/lessons/{lesson_id} endpoint, which is our broken authentication example. Create a file named ch12/tests.py to implement the tests. As illustrated in listing 12.2, we begin by creating an instance of our test client, then we create a function named test_broken_authentication(), which implements the test. To run the test, we need a course ID and a lesson ID, which we fetch directly from the database using our session factory object, session_maker(). Once we have collected this information, we send an unauthenticated request to the endpoint, and we expect to get a 403 (Forbidden) status code.

The right response code in this situation is a 401 (Unauthorized) status code, which means that the request lacks valid authentication credentials, while 403 (Forbidden) means that the request has valid authentication credentials, but the user isn’t allowed to access the requested resource or operation. However, due to a bug in FastAPI (https://github.com/fastapi/fastapi/issues/10177), the server returns a 403 in these situations. There’s an open pull request on GiHub as of now to fix this issue (https://github.com/fastapi/fastapi/pull/13786), so by the time you’re reading this, the framework might be returning the right status code.

Listing 12.2 Running a broken authentication test

# file: ch12/tests.py
from sqlalchemy import select
from starlette_testclient import TestClient

from db_session import session_maker
from models import Course
from server import server

client = TestClient(server)    #A

def test_broken_authentication():
    with session_maker() as session:    #B
        course = session.scalar(select(Course))    #C
        lessons = course.lessons
    response = client.get(
        f"/courses/{course.id}/lessons/{lessons[0].id}"
    )    #D
    assert response.status_code == 403    #E

To run the test in listing 12.2, we’ll use Pytest, Python’s most popular test execution framework. Run the following command to install pytest as a development dependency:

uv add –dev pytest

To run the test, execute the following command:

PYTHONPATH=`pwd` pytest tests.py

The PYTHONPATH environment variable tells Python where to import from, which in this case is from the present working directory. We do this to make our imports from local files in tests.py, such as models and server, work. If you run the above command, you’ll see that it comes back with an assertion error saying that the API didn’t respond with the expected status code. This is excellent: we just discovered a major security hole in our API with a very simple test. This test also covers one of the threat models from section 12.1: unauthorized access to course content. We are starting to take meaningful steps towards improving our security posture. Can you figure out how to fix the implementation to make the test pass? Try to come up with the solution on your own, and check out the ch12/fixed_server.py file in the book’s code repository for a full solution to this problem.

Now that we know how to write broken authentication tests, let’s step up the game by adding a broken object-level authorization (BOLA) test. We’ll run this test against the GET /registrations/{registration_id} endpoint. This endpoint returns the details of a course registration, including full personal details of the student. Since the endpoint returns such sensitive data, we want to make sure users can’t access each other’s course registration details, so let’s write a test for that.

Listing 12.3 illustrates how we write a BOLA test for the GET /registrations/{registration_id} endpoint. The code in listing 12.3 goes in ch12/tests.py where we previously added the broken authentication test, therefore it only shows the new additions.

We begin by fetching a student record from the database and a registration that doesn’t belong to that student. We also create a dummy access token for a test subject and use it to authenticate our request to the GET /registrations/{registration_id} endpoint. Since we are requesting data that belongs to another user, we expect the API to respond with a 403 status code. The 403 (Forbidden) status code means that the API has valid credentials but the user doesn’t have permission to access the requested data or operation, hence it’s the right response status code in this situation.

Listing 12.3 Implementing a BOLA test

# file: ch12/tests.py
[...]
from models import Course, CourseRegistration, Student
[...]

client = TestClient(server)
[...]

def test_bola():
    with session_maker() as session:
        student = session.scalar(select(Student))
        registration = session.scalar(
            select(CourseRegistration).where(
                CourseRegistration.student_id != student.id
            )
        )
    token = create_access_token(sub="test")    #A
    response = client.get(
        f"/registrations/{registration.id}",
        headers={    #B
            "Authorization": f"Bearer {token}"
        }
    )
    assert response.status_code == 403

To run the test in listing 12.3, execute the following command from your terminal:

PYTHONPATH=`pwd` pytest tests.py -k "bola"

The -k flag instructs pytest to only run tests containing the keyword contained in the string passed as a value. If you run the test, you’ll see that it fails, since the endpoint is not enforcing use-based access controls correctly. This is great - we just saved the day again with another simple test! BOLA is one of the most common vulnerabilities found in APIs, so it really pays off to sit down and write for every endpoint that applies user-based access controls.

The final test we are going to implement in this section is a privilege escalation test. Our API has an admin endpoint, which is GET /admin/students, and we want to ensure that only admin users can access the endpoint. Listing 12.4 shows how the test works. We begin by obtaining an access token for a test user without any additional privileges, and we use that token to send a request to the GET /admin/students endpoint. We expect the endpoint to respond with a 403 (Forbidden) status code since the access token doesn’t have admin privileges. Like in the BOLA scenario, the request contains valid credentials, but the user does not have permission to access the requested resource or operation, hence the 403 status code is the right choice for this situation.

Listing 12.4 Testing privilege escalation attacks

# file: ch12/tests.py
[...]
def test_rbac():
    token = create_access_token(sub="test")
    response = client.get(
        f"/admin/students",
        headers={"Authorization": f"Bearer {token}"}
    )
    assert response.status_code == 403

To run the test in listing 12.4, use the following command:

PYTHONPATH=`pwd` pytest tests.py -k "rbac"

If you run the test, you’ll see that it fails, since the API doesn’t enforce role-based access controls correctly and hence responds with a 200 when a student attempts to access an admin endpoint. Privilege escalation is one of the most severe types of vulnerabilities in cybersecurity, and it’s really great that we were able to catch this flaw with a simple test like the one in listing 12.4. This test also addresses one of our threat models from section 12.1: unauthorized access to personal data via the admin portal. I encourage you to create similar tests for your own APIs, and as an exercise, figure out how to fix the privilege escalation vulnerability in the GET /admin/students endpoint.

The tests we have written in this section are simple yet very powerful since they target critical vulnerabilities that are common in many APIs. These are the low-hanging fruits that yield the highest return to our efforts. However, our job isn’t done yet. The next step is to create tests that involve exploiting complex business flows, and that’s the topic of the next section!

12.5 Testing business flow vulnerabilities

A growing number of API breaches happen when threat actors exploit vulnerable business flows. As we discussed in section 4.6, business flow exploits are very difficult to mitigate using runtime threat detection and protection tools. While attacks like SQL injection, XSS, and brute force bear a clear footprint that most web application firewalls can identify and block, business flow exploits are more elusive, since they are intimately related to the business logic behind the application. Without a clear understanding of how your application works, standard tools are blind to business flow exploits. Therefore, it’s imperative that we spend time threat modeling business flow exploits in our APIs, and that we write tests to check if we are vulnerable to them. That’s what we are going to do in this section!

As discussed in section 12.1, our threat modeling exercise has identified a business flow exploit that we want to prevent: abuse of coupon codes. When a student registers with our website, we issue a one-time 80% discount code that they can use to sign up for a course. We do this to incentivize students to sign up with our platform and start consuming our content. However, we also want to ensure students only use coupon codes they have legitimate access to, since abusing them will put our business at stake. Our threat model considers two scenarios:

Students using the same coupon code multiple times

Students using discount codes from other students

We want to write unit tests to make sure none of these things happen. The tests we are going to write in this section require more setup and data compared to the tests we wrote in section 12.4, so we are going to create a few pytest fixtures to generate such data. Listing 12.5 adds two fixtures: one, named new_student_voucher(), which generates a newly registered student ID and their voucher code, and another one named course_id(), generates a course ID. We will use both fixtures in our flow tests. As you can see, pytest fixtures are simple Python functions that return the values we need in our tests, and they are decorated with pytest’s fixture() decorator.

Listing 12.5 Creating fixtures to set up tests with pytest

# file: ch12/tests.py
[...]
import pytest    #A
[...]

@pytest.fixture    #B
def new_student_voucher():
    student_details = {
        "first_name": "John",
        "last_name": "Connor",
        "date_birth": datetime(1985, 2, 28),
        "phone_number": "634-430-6045",
        "address": "19828 Valerio Street, Winnetka, CA",
        "email": "john.connor@apithreats.com",
        "password": "john.connor",
    }
    with session_maker() as session:
        student, voucher = register_new_student(
            student_profile=RegisterStudent(**student_details),
            db_session=session
        )    #C
        return student.id, voucher.code    #D

@pytest.fixture
def course_id():
    with session_maker() as session:
        course = session.scalar(select(Course))
        return course.id    #E

Now that we have the fixtures, we need to set up the preconditions for our tests. Let’s write them! We’ll begin with the coupon reuse test, which is implemented in listing 12.6. As you can see, we use the fixtures we implemented in listing 12.5 by simply adding them to the test’s function signature. Pytest takes care of calling those fixture functions and fetching their values for our test. We create an access token for the newly registered student, and we put together the payload needed to register the student for a course using their coupon code. Then, we attempt to register the student twice using the same coupon. For the first registration, we expect the API to respond with a 201 (Created) status code, which means that the operation was successfully completed. However, for the second attempt, we expect a 409 (Conflict) response. The 409 status code means that the requested operation cannot be processed since it conflicts with the current state of the resource. In this case, the 409 response indicates that the coupon code has already been used and therefore cannot be claimed again.

Listing 12.6 Testing coupon reuse

# file: ch12/tests.py
[...]
def test_coupon_code_reuse(    #A
    new_student_voucher, course_id
):
    student_id, voucher_id = new_student_voucher    #B
    token = create_access_token(    #C
        sub=str(student_id)
    )
    registration_payload = {
        "voucher": str(voucher_id),    #D
        "card_number": "5352 6878 8810 6793",
        "cvv": 369,
        "expiry_date": "01/28",
    }
    first_registration = client.post(
        f"/courses/{course_id}/register",    #E
        headers={    #F
            "Authorization": f"Bearer {token}"
        },
        json=registration_payload,
    )
    second_registration = client.post(    #G
        f"/courses/{course_id}/register",
        headers={"Authorization": f"Bearer {token}"},
        json=registration_payload,
    )
    assert first_registration.status_code == 201    #H
    assert second_registration.status_code == 409    #I

To run the test in listing 12.6, use the following command:

PYTHONPATH=`pwd` pytest tests.py -k "coupon"

If you run the test, you’ll see that it fails since the API returns 201 status codes to both requests, meaning it is not checking whether a coupon code has already been used. This is a major business flow vulnerability in our API, and we need to fix it lest students exploit it at the cost of our business. Can you figure out how to fix this vulnerability? Try to come up with the solution on your own, and check out the ch12/fixed_server.py file in the book’s code repository for a full solution to this problem.

The second scenario we want to test is when students try to use coupon codes from other students. Students could exploit such a flaw to register tons of new phantom accounts only to use their coupon codes through their main account. Listing 12.7 shows how we put together a test for this scenario. The test uses the two fixtures from listing 12.5 to obtain the voucher of a newly registered student and a course ID. We also fetch from the database a formerly registered student (i.e., a student whose ID is different from the newly registered student), and issue an access token for them. Finally, we send a request to the API using the former student’s access token and the newly registered student’s coupon code, and we expect the API to respond with a 403 (Forbidden) status code, since we are using a coupon code that doesn’t belong to us.

Listing 12.7 Testing business flow attacks

# file: ch12/tests.py
def test_coupon_abuse(new_student_voucher, course_id):
    new_student_id, new_voucher_id = new_student_voucher
    with session_maker() as session:
        student = session.scalar(
            select(Student).where(    #A
                Student.id != new_student_id
            )
        )
        token = create_access_token(    #B
            sub=str(student.id)
        )
    registration_payload = {
        "voucher": str(new_voucher_id),    #C
        "card_number": "5352 6878 8810 6793",
        "cvv": 369,
        "expiry_date": "01/28",
    }    registration = client.post(
        f"/courses/{course_id}/register",
        headers={"Authorization": f"Bearer {token}"},
        json=registration_payload,
    )
    assert registration.status_code == 403

To run the test in listing 12.7, use the following command:

PYTHONPATH=`pwd` pytest tests.py -k "coupon_abuse"

If you run the test, you’ll see that it fails since the API isn’t enforcing user-based access controls on coupon codes, and hence returns with a 201 status code to our test in listing 12.7. As is usually the case with business flow exploits, the flaw in this case involves other types of vulnerabilities, such as a BOLA exploit on the coupon code itself. Fixing this type of vulnerability is crucial for the sustainability of our business – can you figure out how to apply the right fix and get the test passing green? Try to come up with the solution on your own, and check out the ch12/fixed_server.py file in the book’s code repository for a full solution to this problem.

This concludes our journey through automated API security testing! If you follow the recommendations in this chapter, you’ll be in an excellent position to discover and prevent tons of vulnerabilities in your APIs. And since these are unit tests, you’ll be able to run them on every code change to check that your APIs remain secure. Even if you don’t find proper vulnerabilities, you will likely find that your API behaves in unexpected ways when subjected to the test strategies outlined in this chapter, and threat actors know how to take advantage of such odd behaviors, so make sure you address them all!

12.6 Summary

To improve the security profile of our APIs, we must lay out a continuous security testing strategy that helps us detect vulnerabilities with every code change. A good security testing strategy begins with our threat models and helps us prioritize tests that are easier to write and address the biggest and most widespread vulnerabilities in our codebase.

The first step to assess the security profile of our APIs is to check for design vulnerabilities. We do this by running API linters, like Spectral, which analyze our API specification for flaws in our endpoints and input parameter definitions.

Once we have a solid API design, we ensure that the implementation conforms to it. We do this by using contract testing tools like schemathesis that check how our API behaves when presented with random malicious and malformed payloads.

Standard tools like Spectral and schemathesis don’t test our business layer, and for better coverage, we need unit tests that reach that layer in our applications. A good starting point is writing tests around access controls, such as broken authentication, broken object-level authorization, privilege escalation, and attribute-based access controls.

In addition to targeting our business layer, we also want to write tests around our business flows. Our threat models may reveal situations where users can perform a sequence of actions that can lead to an exploit, and we want to write unit tests for such scenarios too. Such tests are complex and require more setup, but they also yield the highest benefits.

Appendix A. API security checklist

A common concern among API developers is whether they’ve done everything they can to ensure their APIs are secure, and many developers ask for a checklist that allows them to determine if there’s anything else left to do. This appendix provides such a checklist. As you’ve learned throughout the book and is clear from this appendix, security cannot be a last-minute concern in your API development process. If you’re serious about API security, you must build security into your APIs by shifting your security efforts to the left; that is, to the beginning of the API development process. That’s why the first section of this appendix deals with API design and follows with implementation, and finally, authentication and authorization, infrastructure, and observability. The goal is to build security into each of these building blocks. Each section of the appendix maps to one or more chapters of the book, as indicated.

A.1 Design

Chapters 2, 3, and 6.

Consolidate your API design an interface description language (IDL) like OpenAPI, Schema Definition Language (SDL), etc.

Test the design for security vulnerabilities (e.g. with Spectral).

Constrain every input schema and parameter.

Threat model the design.

Describe your expected user flows using a standard format (Arazzo) and threat model the flows.

A.2 Implementation

Chapters 4, 5, and 12.

Use a robust API development framework (e.g. FastAPI for Python, Micronaut for Java, etc.).

Use secure implementation patterns (e.g. data transfer objects, object relational mappers, parametrize database queries, etc.).

Validate everything (incoming data, including from third-party integrations, outgoing data).

Validate the implementation against the specification using an API fuzzer (e.g. Schemathesis).

Use an outbound proxy to handle outbound requests securely and handle failing external dependencies gracefully.

Write unit tests to check if the API is vulnerable to your threat models.

A.3 Authentication and authorization

Chapters 7, 8, and 10.

Use an identity provider like Auth0, Authlete, Curity, AWS Cognito, etc. If that’s not possible, self-host an identity provider like Keycloak.

Use the right Open Authorization (OAuth) flow for your use case.

Use standard JSON Web Tokens (JWTs) with robust signatures (preferably PS256 or similar). Rotate your signing keys often.

Don’t use ID tokens as access tokens

Use robust libraries to validate access tokens.

If your API handles highly sensitive data, use sender-constrained tokens.

Require authentication on every endpoint by default. Explicitly mark those endpoints that don’t require authentication.

Define clear user roles and apply role-based access controls consistently across all endpoints.

A.4 Infrastructure

Chapter 9.

Use an API Gateway as your single entry point for all incoming traffic. Use the API Gateway to manage your inventory of APIs and keep your attack surface under control.

Deploy a web application firewall (WAF) to protect against DoS and other attacks.

Segment your network where services don’t need to talk to each other.

Implement rate-limiting policies that adapt to your user behavior and can vary according to the sensitivity of every endpoint and flow.

A.5 Observability

Chapter 11.

Create logs, traces, and metrics for every relevant event in your system.

Track the origin of every request, including session ID, user account, geolocation, IP, and device fingerprint.

Study user behavior to understand how users engage with your APIs and adapt your threat detection models and rate-limiting policies.

Deploy threat-detection solutions to identify business flow exploits in real time.

Appendix B. Setting up Auth0 for authentication and authorization

This appendix walks you through the process of setting up and configuring Auth0 to handle authentication and authorization for your APIs. Specifically, we will

Register an API, which we will call “Orders API.”

Register a client that uses the authorization code flow to allow users to log in and use the API.

Create an admin role for the API.

This setup is necessary if you want to be able to code along with the practical examples in chapter 8. After reading this appendix, you’ll be ready to configure Auth0 for your own applications and projects. If you wish you use complimentary video material to this appendix, I have created a series of videos on this topic that you will find useful:

“Setting up Auth0 for API authentication and authorization” (https://youtu.be/PbUcQUQ7K2o).

“Login and issue API access tokens with Auth0 and FastAPI” (https://youtu.be/ato2S5b27o8).

“Validate JWTs issued by Auth0 in FastAPI” (https://youtu.be/AtmyC945_no).

To get started, go to Auth0’s website (https://auth0.com/) and create an account. When you create an account, you get assigned a tenant by default. You can follow this guide with the default tenant, or you can create a new tenant if you wish. When you create a new tenant, you select the region where you want to create the tenant, select a unique domain for your tenant, and choose the tenant’s environment. The choices for environment are development, staging, and production. For this guide, choose the development environment, which is ideal for local development, testing, and getting familiar with Auth0. An Auth0 production environment has production-grade protection and security requirements, so it won’t let you use localhost URLs in your configuration, and it will prevent you from attempting multiple logins when you’re testing and trying to solve an error, which will get in the way of your debugging process.

Once you are in your tenant’s page, click on Applications in the right sidebar, and then select APIs from the dropdown. As illustrated in figure B.1, you’ll be directed to the tenant’s APIs page, where you’ll see an API already registered named Auth0 Management API. This is the user management API, which comes by default with all tenants, and it allows you to manage your users programmatically.

Figure B.1 In your tenant’s APIs page, you can register APIs for which you want to handle authentication and authorization with Auth0.

On the top-left corner, click “Create API” to register your first API. As illustrated in figure B.2, this action prompts you with a pop-up form to fill in some configuration details about the API. Here’s how to fill in the form:

Name: a human-friendly name for the API such as “Orders API.”

Identifier: a machine-friendly identifier for the API. The value can be anything you want, but for convenience (and for sanity), I recommend you give it a meaningful identifier for your API. For example, http://dev.apithreats.com/api/orders clearly identifies the orders API in the dev environment for apithreats.com.

JSON Web Token (JWT) Profile: you can choose to use RFC 9068 (https://www.rfc-editor.org/rfc/rfc9068.html) tokens or Auth0 tokens. Both are very similar, with the main difference being that RFC 9068 tokens include the jti (JWT ID) claim while the Auth0 tokens do not, and Auth0 tokens include the azp (client ID) claim, which RFC 9068 tokens do not. For this guide I’ve selected the Auth0 profile, but Auth0 tokens will work just as well. You can learn more about the differences between the two profiles here: https://auth0.com/docs/secure/tokens/access-tokens/access-token-profiles.

JSON Web Token (JWT) Signing Algorithm: the signing algorithm you want to use for your JWTs. In the free plan, you can choose between HS256 and RS256. As you learned in chapter 7, asymmetric signing keys are more secure, so we choose RS256.

Figure B.2 When you register an API, you are asked to provide a human-friendly name, an identifier, and select the JWT profile and signing algorithm you want to use for your access tokens.

Once you’ve configured the API, click on the Create button in the pop-up panel, and you will be directed to the API’s configuration page. If you don’t get directed to the API’s configuration page, you can get there by selecting “Applications” in the right sidebar, then click on APIs, and finally selecting the API we just created.

On the API’s configuration page, select the Settings tab. As shown in figure B.3, the Settings page contains all the sensitive configuration we need to integrate our APIs with Auth0. Scroll down this page until you arrive at the RBAC Settings section, and in that section, toggle on Enable RBAC and Add Permissions in Access Token. This setting will allow us to create different access permissions for the API and to include those permissions in the access token so we can easily validate them on our server.

Figure B.3 In the API’s settings page, you configure how you want to allow clients to access the API. For example, you can configure how long access tokens are valid for and whether the API accepts role-based access controls.

After configuring RBAC settings, scroll down to the Access Settings section and toggle on Allow Offline Access. This setting allows you to request refresh tokens from Auth0 during the authorization request. If you don’t turn on this feature, Auth0 will not issue refresh tokens for your API. Once we’ve done all this, scroll down to the bottom of the page and click on the Save button to persist the current configuration.

Now that our API is configured, let’s create some permissions. As we mentioned at the beginning of the Appendix, we are going to create an admin role, so let’s create an access permission scope for this API. Go to the top of the API configuration page and click on the “Permissions” tab, as illustrated in figure B.4. On the permissions screen we can add as many access scopes as needed for the API. Auth0 suggests adding granular scopes like “read:orders” and “create:orders”. Such scopes are very granular and give access to very specific operations which can later be aggregated and consolidated into a role. For the purpose of this guide, let’s create a permission named “admin” and add the following description: “Admin role.” After doing this, click the + Add button to add the permission to the API.

Figure B.4 In the API’s permissions page, we can configure access scopes for the API.

Now that we have an API with a permission access scope, let’s register a client to access the API. On the right sidebar, click on “Applications” and then select “Applications” from the dropdown, as shown in figure B.5. This action takes you to your tenant’s applications page, where you will see two applications: Default App and Orders API (Test Application). Orders API (Test Application) was automatically created by Auth0 when we registered our API; and it's a machine-to-machine client that uses the client credentials flow and it allows us to run a quick test against the API to ensure it’s working correctly.

Let’s register a client that allows us to access the API using a web application. Click on the + Create Application button on the top left corner of the applications screen. Give the application a meaningful name. Since we are going to use this client in a web application, a useful name is “Orders web application client.” After naming the application, we have to select the type of application we want to register. We have the following choices of application types:

Native: for mobile applications, TV applications, etc. This type of client uses the authorization code flow with PKCE.

Single Page Applications: this choice is suitable if we plan to run the authorization flow entirely from the browser. Within this option, Auht0 allows you to use the implicit flow and the authorization code flow with PKCE. As you learned in chapter 7, the implicit flow is no longer recommended by the latest OAuth specifications, so if you choose this option, make sure you use the authorization code flow with PKCE.

Regular web applications: this is the choice we are going to select in this guide. This option is suitable for traditional web applications with content dynamically rendered in the server, but it works for single page applications too. The only flow available for this type of application is the authorization code flow, which we can run with PKCE optionally too.

Machine to Machine Applications: this option is flow clients that use the client credentials flow.

For this guide, please select the Regular Web Applications option since we’ll use the authorization code flow to authorize the client. After selecting the application type, you have the option to download ready-made boilerplate code to handle authentication in our applications. Chapter 8 walks you through all the steps to authorize a client, so you can skip this step and go straight to the Settings tab, as illustrated in figure B.5.

Figure B.5 In the application’s settings page, you have all the information you need to integrate your clients with Auth0, including your tenant’s domain and the client’s ID and secret.

As illustrated in figure B.5, the application’s settings page contains all the information we need to integrate our client with Auth0. The Basic Information section contains our tenant’s domain. Using this domain, we can obtain our tenant’s OIDC discovery endpoint, which is available under https://<your-tenant-domain>/.well-known/openid-configuration. For example, for apothreats.com, which uses Auth0 to handle authentication and authorization, the OIDC discovery endpoint is https://apithreats.eu.auth0.com/.well-known/openid-configuration. Using your tenant’s domain, you can also derive the authorization URL like this: https://<you-tenant-domain>/.authorize.

The application’s settings page also includes the application’s client ID and client secret. These are the values you need to replace for the CLIENT_ID and the CLIENT_SECRET parameters in the listings in chapter 8.

Within the application’s settings page, scroll down to the Application URIs section, as illustrated in figure B.6. Within the “Allowed Callback URIs” field, we include the URLs we want to be able to redirect the user to after successfully logging in with Auth0. This is the value that goes into the request_uri parameter in the authorization request URL. To follow along with the examples in chapter 8, include the following URL in this section: http://localhost:8000/docs, which is the URL we redirect users back to after login.

Figure B.6 In the “Application URIs” section within the application’s settings page, you configure the allowed callback URLs, which are the URLs you want to redirect users to after successfully logging in with Auth0.

After configuring the callback URLs, scroll down to the section named Refresh Token Rotation and toggle on Rotation to configure Auth0 to expire our refresh tokens and issue new ones after being used. Without sender-constrained tokens, this is a good measure to prevent token replay attacks. After applying all this configuration, click the Save Changes button on the bottom-right corner of the screen. Our client is now ready to use!

The final step to set up our Auth0 tenant is creating the admin role. On the right sidebar, click on User Management and then select Roles from the dropdown. In the roles page, click on Create Role, which brings up a pop-up panel to configure our role, as shown in figure B.7. Since we are creating an admin role, let’s name the role “admin” and include the following description: “Admin role.” Then click Create to create the role. That action takes us to the role’s configuration page.

Figure B.7 When you create a new role, you are asked to provide the role’s name and description.

On the role’s configuration page, select the Permissions tab. This screen allows us to attach permissions from our APIs to the role. Click on Add Permissions, which brings up a pop-up panel from which we can select the permissions we want to attach to the role, as illustrated in figure B.8. In the pop-up panel, select “Orders API,” and then click on the “admin” permission to attach it to the role. Finally, click the Add Permissions button to persist the configuration.

Figure B.8 To attach access scopes to a role, we add permissions from our previously created APIs.

Now that we have an admin role available, we can assign it to users. To do this, you’ll need to first register a user. Follow the steps in chapter 8 to handle authentication with Auth0. When you initiate the authorization request, you’ll need to log in with Auth0, and if you don’t have an account, you’ll be asked to create one first. Once the account is created, you can go back to your tenant’s management dashboard. In the right sidebar, click on User Management, and then select Users from the dropdown to go to the user management dashboard. Once there, click on the user account you want to promote to admin. As illustrated in figure B.9, this action takes you to a page where you can see all the details about the user. On this page, you can assign permissions and roles to the user. If you click on the Permissions tab, you’ll be able to assign specific access scopes from your APIs. If you click on the Roles tab, you’ll be able to assign a role. Once you’ve taken any of these actions, the permissions will be inserted in the access token in a custom field named permissions. You can also automate the processing of adding roles and permissions to users by using Auth0’s management API.

Figure B.9 In the permissions tab within the user’s management page, we can assign roles and permissions to the user.

We are done! You are ready to set up a tenant in Auth0, register APIs and clients to handle authentication and authorization, and create roles and permissions to handle role-based access controls.

This concludes our guide to setting up and configuring Auth0 to handle authentication and authorization for our APIs. You’re ready to roll out your own secure APIs!

Appendix C. API security RFCs and learning resources

A great deal of API security is dealing with standards and protocols. Those standards and protocols are described in formal documents called Request for Comments (RFCs), which are published by the Internet Engineering Task Force (IETF). Throughout this book, we have made references to many RFCs that describe how Open Authorization works, what JSON Web Tokens look like, what are JSON Web Keys, and so on. And by this point, you may be wondering, “damn, where is that RFC that describes what JSON Web Keys are?” Well, wonder no more. In this appendix, I’ve put together the most important RFCs that you, as an API security practitioner, should know about, and I highly encourage you to read through them.

It goes without saying that you cannot cover everything there is to know about API security in one book. This book gives you a very solid foundation, but you will want to go deeper into other topics such as threat modeling and API design. Also, this book approaches API security from the builder’s perspective, but what about the hacker’s perspective? Learning how threat actors think and abuse vulnerabilities will help you get better at protecting your own APIs. The second part of this appendix lists some resources you can use to take your journey learning more about API security to the next level.

C.1 Important RFCs

The development of RFCs relevant for API security is currently overseen by IETF’s Web Authorization Protocol (OAuth) Working Group. The full list of RFCs is large and you can check it out on this website: https://datatracker.ietf.org/wg/oauth/documents/. Below is a list of the most important RFCs grouped by sections.

C.1.1 JSON Web Tokens (JWT)

RFC 7515: M. Jones, J. Bradley, and N. Sakimura , “JSON Web Signature (JWS),” RFC 7515 (May 2015, https://www.rfc-editor.org/rfc/rfc7515.html).

RFC 7516: M. Jones and J. Hildebrand, “JSON Web Encryption (JWE),” RFC 7516 (May 2015, https://www.rfc-editor.org/rfc/rfc7516.html).

RFC 7519: M. Jones, J. Bradley, and N. Sakimura, “JSON Web Token (JWT),” RFC 7519 (May 2015, https://datatracker.ietf.org/doc/html/rfc7519).

RFC 8725: Y. Sheffer, D. Hardt, and M. Jones, “JSON Web Token Best Current Practices,” RFC 8725 (Feb 2020, https://datatracker.ietf.org/doc/html/rfc8725).

RFC 7517: M. Jones, (“JSON Web Key (JWK)”, RFC 7517 (May 2015, https://datatracker.ietf.org/doc/html/rfc7517).

RFC 7518: M. Jones, “JSON Web Algorithms (JWA),” RFC 7518 (May 2015, https://www.rfc-editor.org/rfc/rfc7518.html).

C.1.2 Open Authorization (OAuth)

RFC 5849: E. Hammer-Lahav, “The OAuth 1.0 Protocol,” RFC 5849 (April 2010, https://datatracker.ietf.org/doc/html/rfc5849).

RFC 6749: D. Hardt, “The OAuth 2.0 Authorization Framework,” RFC 6749 (October, 2012, https://www.rfc-editor.org/rfc/rfc6749.html).

RFC 9700: T. Lodderstedt, J. Bradley, A. Labunets, and D. Fett, “Best Current Practice for OAuth 2.0 Security,” RFC 9700 (January 2025, https://datatracker.ietf.org/doc/html/rfc9700).

draft-ietf-oauth-v2-1: D. Hardt, A. Parecki, T. Lodderstedt, “The OAuth 2.1 Authorization Framework,” draft-ietf-oauth-v2-1 (November 2024, https://datatracker.ietf.org/doc/draft-ietf-oauth-v2-1/).

RFC 7636: N. Sakimura, J. Bradley, and N. Agarwal, “Proof Key for Code Exchange by OAuth Public Clients”, RFC 7636 (September 2015, https://www.rfc-editor.org/rfc/rfc7636).

RFC 8628: W. Denniss, J. Bradley, M. Jones, H. Tschofenig, “OAuth 2.0 Device Authorization Grant,” RFC 8628 (August 2019, https://datatracker.ietf.org/doc/html/rfc8628)

RFC 6819: T. Lodderstetdt, M. McGloin, and P. Hunt, “OAuth 2.0 Threat Model and Security Considerations,” RFC 6819 (Janunary 2013, https://datatracker.ietf.org/doc/html/rfc6819).

RFC 8705: B. Campbell, J. Bradley, N. Sakimura, and T. Lodderstedt, “OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens,” RFC 8705 (February 2020, https://datatracker.ietf.org/doc/html/rfc8705).

RFC 9449: D. Fett, B. Campbell, J. Bradley, L. Lodderstedt, M. Jones, and D. Waite, “OAuth 2.0 Demonstrating Proof of Possession (DPoP),” RFC 9449 (https://datatracker.ietf.org/doc/html/rfc9449).

RFC 9101: N. Sakimura, J. Bradley, and M. Jones, “The OAuth 2.0 Authorization Framework: JWT-Secured Authorization Request (JAR),” RFC 9101 (September 2021, https://www.rfc-editor.org/rfc/rfc9101).

RFC 9126: T. Lodderstedt, B. Campbell, N. Sakimura, D. Tonge, and F. Skokan, “OAuth 2.0 Pushed Authorization Requests,” RFC 9126, (September 2021, https://www.rfc-editor.org/rfc/rfc9126).

RFC 7662: J. Richer, “OAuth 2.0 Token Introspection,” RFC 7662 (October 2015, https://www.rfc-editor.org/rfc/rfc7662.html).

RFC 9701: T. Lodderstedt and V. Dzhuvinov, “JSON Web Token (JWT) Response for OAuth Token Introspection,” RFC 9701 (January 2025, https://www.rfc-editor.org/rfc/rfc9701.html).

draft-ietf-oauth-rar: T. Lodderstedt, J. Richer, B. Campbell, “OAuth 2.0 Rich Authorization Requests,” draft-ietf-oauth-rar (August 2023, https://datatracker.ietf.org/doc/html/draft-ietf-oauth-rar)

C.1.3 OpenID Connect (OIDC)

N. Sakimura, J. Bradley, M. Jones, B. de Medeiros, C. Mortimor, ” OpenID Connect Core 1.0,” (December 2023, https://openid.net/specs/openid-connect-core-1_0.html).

T. Lodderstedt, S. Low, D. Postnikov, “Grant Management for OAuth 2.0,” (May 2025, https://openid.bitbucket.io/fapi/oauth-v2-grant-management.html).

C.1.4 FAPI

D. Fett, D. Tonge, J. Heenan, “FAPI 2.0 Security Profile,” (May 2025, https://openid.bitbucket.io/fapi/fapi-security-profile-2_0.html).

D. Fett, D. Tonge, J. Heenan, “FAPI 2.0 Message Signing,” (March 2025, https://openid.bitbucket.io/fapi/fapi-2_0-message-signing.html).

D. Fett, “FAPI 2.0 Attacker Model” (December 2022, https://openid.net/specs/fapi-2_0-attacker-model-ID2.html) .

Pedram Hosseiyni, Ralf Küsters, and Tim Würtele, “Formal Security Analysis of the OpenID FAPI 2.0: Accompanying a Standardization Process,” 2024 IEEE 37th Computer Security Foundations Symposium (CSF) (2024, pp. 589-604, https://doi.ieeecomputersociety.org/10.1109/CSF61375.2024.00002).

C.2 Learning resources

APISec University courses (https://www.apisecuniversity.com/).

Ball, Corey J., Hacking APIs (No Starch Press, 2022).

Domoney, Colin, Defending APIs Against Cyber Attack (Pack, 2024).

Farhi, Dolev and Nick Aleks, Black Hat GraphQL (No Starch Press, 2023).

Haro Peralta, José, Microservice APIs (Manning, 2023).

Lauret, Arnaud, The Design of Web APIs, second edition (Manning, 2025).

Madden , Neil, API Security in Action (Manning, 2020).

Ponelat, Joshua S. and Lukas L. Rosenstock, Designing APIs with Swagger and OpenAPI (Manning, 2022).

Richardson, Chris, Microservices Patterns (Manning, 2018).

Richer, Justin and Antonio Sando, OAuth 2 in Action (Manning, 2017).

Riedesel, Jamie, Software Telemetry (Manning, 2021).

Shostack , Adam, Threat Modeling (Wiley, 2014).

Siriwardena, Prabath, Advanced API Security in Action (Apress 2019).

Siriwardena, Prabath and Nuwan Dias, Microservices Security in Action (Manning, 2020).

Wong, David, Real-World Cryptography (Manning, 2021).