Privacy-Enhancing Technologies for Artificial Intelligence-Enabled Systems (2024)

1.1. Data in Use for AI-Enabled Systems

The term ”data in use” refers to any data that is actively being processed (created, accessed, updated, or erased) by a human or a machine. In the context of AI, a human process could be performing data analysis; a machine process could be model training or performing model inference.

Attacks against data-in-use processes at large have been well researched, (Pitropakis etal., 2019; Dwork etal., 2017) and these general attacks certainly pose a threat to AI-enabled systems. However, AI-enabled systems also have their own unique set of potential attack vectors, such as those described below.

Ideally, AI models will only be run by model owners, as explicitly permitted by data owners, in a way that guarantees the data owner’s privacy. This is difficult, because AI models have the potential to be attacked at any point in the model’s lifecycle: data set creation, model development, model deployment, user access, or user execution. Protections need to be put in place at each of these stages to ensure the system does not violate data owner privacy.

1.2. Privacy and Confidentiality

Not all sensitive data is the same. Level of sensitivity and, consequently, level of protection required differ depending on specific use case and associated legal standards. These distinctions are important because they lead to different requirements for a system’s architecture. Two important, high-level concepts when dealing with sensitive data are privacy and confidentiality.

The primary difference between privacy and confidentiality is that the former concerns personal information and the latter concerns any data considered sensitive. Moreover, confidentiality concerns the unauthorized use of information already in the hands of an organization, (National Cybersecurity Centerof Excellence, nd) whereas privacy concerns the rights of an individual to control the information that an organization collects, uses, and shares with others.

Of particular concern is Personally Identifiable Information (PII), ”[a]ny representation of information that permits the identity of an individual to whom the information applies to be reasonably inferred by either direct or indirect means.” (ofLabor, nd) PII data has legal, contractual, and ethical requirements that restrict its disclosure, and its use is controlled by data privacy regulations (e.g., HIPAA (ofHealth and Services, nd), GDPR (Wolford, nd)).

1.3. Privacy-Enhancing Technologies (PETs)

PETs are technologies that aim to protect data-in-use processes without preventing the system from accomplishing any of its necessary functions. Specifically, PETs are designed to do the following:

All of the aforementioned tasks are concerned with the protection of sensitive data and addressing data layer vulnerabilities. In practice, the term “privacy-enhancing technologies” encompasses a wide variety of tools – whether implemented via hardware or software, on premises or in the cloud – that are all designed to protect data-in-use processes. Figure 1 organizes various PETs by their primary characteristics and demonstrates the breadth of tools and functionalities that fall under the “PETs” umbrella.

Multiple PETs may operate in tandem to address a particular security concern or set of security concerns more fully. As Figure 2 shows, each PET has its own strengths and weaknesses; there is no “one size fits all” PET.

The columns in the figure represent system-level characteristics that could be affected by the addition of a given PET. The primary capabilities of each PET are shown in this chart as a “Perfect Fit.” The “Potentially Viable” assignment means that, while the PET in its standard form could pose a problem, it can be modified to perform better in that category. The chart also shows why a given PET ought not to be used. For example, fully hom*omorphic encryption (FHE) can be very detrimental to a system’s speed and scalability, so it is classified as a ”Show Stopper” for systems dependent upon those features.

PETs have great potential to improve the security of existing systems. Perhaps more notably, though, they have the potential to enable new systems that were previously not feasible due to data protection concerns. In particular, with PETs, new sources of data can be leveraged while still maintaining PII protections.

2.1. Insider Threats

Insiders are responsible for 55% of cybersecurity incidents. Of this 55%, 57% attack intentionally and 43% act inadvertently. (Bradley, 2015) Insider incidents within non-AI-enabled systems are managed using practices like the principle of least privilege, user and entity behavior analytics (UEBA) (Cavalancia, 2020), and general data loss prevention (DLP) practices (Microsoft, nd). However, the rise of AI-enabled systems offers new ways for insider threats to manifest themselves in an organization.

For example, data leaks can occur when sensitive information is sent to a third-party AI model owner, as seen in Figure 3. A well-known example of a publicly available third-party model is OpenAI’s ChatGPT. (OpenAI, 2022) As it stands, ChatGPT must have access to plaintext prompts to be able to produce responses. This requirement poses a confidentiality concern, since user input is necessarily exposed during ChatGPT’s inference process. Organizations that own models, like OpenAI, thus have direct access to the data sent by users.

The risk of careless insiders unintentionally revealing confidential data to AI-enabled services has led companies such as Samsung (Gurman, 2023) and Northrop Grumman (Ellis, 2023) to establish policies for the careful use of ChatGPT and other publicly available large language models (LLMs). However, the existence of policy does not provide guarantees that employees will follow procedure. Employees still have the ability to inadvertently expose sensitive data to external services without realizing the impact of their actions.

Another unique challenge in the era of AI is that internal model developers now have access to an increased amount of data, some of which may be highly sensitive. For reference, an LLM like ChatGPT required a corpus size of 300 billion words, (Hughes, 2023) whereas previous-generation models, like BERT only required 3.3 billion words. (Devlin etal., 2019) Access to big data opens up the potential for malicious insiders to exfiltrate and release large swaths of information. Careless insiders may also leak these data sets, albeit inadvertently, but it is the concentration of so much data in the hands of potentially malicious people that particularly demands our concern. Furthermore, while malicious insiders may also feed sensitive data to external services like ChatGPT, truly malicious actors will turn their attention toward much higher impact leaks such as model training sets and collected inference data.

Figure 4 shows how data, even data that is secured through standard, recommended security protocols, must normally be decrypted for any form of analysis, putting the data at risk of being distributed. Policy-based solutions, such as least access principles, are also often applied to this problem to minimize access to sensitive data, but few technological methods have been employed to ensure that malicious insiders do not have the ability to gather and distribute model data sets or inject poisoned data into a curated data set.

2.2. Outsider Threats

Outsiders are responsible for the remaining 45% of cybersecurity incidents. (Bradley, 2015) Outsider incidents with non-AI-enabled systems are managed using practices like intrusion detection and prevention systems (IDPS), standard encryption (i.e., in transit and at rest), firewalls, and robust authentication and access control. These work well to defend static data and maintain permissions boundaries. AI-enabled systems work differently though, because the data interaction is not static. Instead of connecting to deterministic end services that have been tightly defined, users are interacting directly with non-deterministic models. This interaction has the potential to be abused through reverse engineering, by untrusted third-party hosts, or as part of an adversarial attack.

A major privacy threat when deploying AI models is that models can potentially be reverse-engineered to reveal sensitive characteristics both about the model itself (Staff, 2021) (which is a problem if the model is proprietary) and about the data that was used to train the model. Those characteristics may even enable someone to re-link the data that was used to create the model to PII data in the model’s training set. This scenario could result in an unintentional release of sensitive data, like in the Netflix Prize, a challenge in which researchers were able to take an anonymized data set and re-link people to their ratings of television shows. (Narayanan and Shmatikov, 2008)

Also, training AI models can be a very computationally intensive process. The training of ChatGPT costs 3,640 PetaFLOP days. (Brown etal., 2020) Thus, compute is regularly outsourced to third parties. If we place sensitive data on third-party servers, we put a high amount of trust that the compute provider will not observe, collect, or tamper with data in unapproved ways.

A final threat posed by outsiders is an adversarial attack. Like any other data, AI models are vulnerable to manipulation, and adversarial attacks manipulate models by sending them examples that are specifically crafted to fool the model (see Figure 5). For instance, researchers highlighted the impact of adversarial attacks by confusing a Tesla Model S and forcing it to moving into oncoming traffic. (Hao, 2019)

2.3. Summary of Risks

A typical, deployed AI-enabled system concerns two types of personnel, system owner and system user, and both are subject to privacy risks.

The owner of the system seeks to protect both the AI model and the data that was used to train it. Protecting the model means keeping its characteristics private, but it also includes maintaining the integrity of the model and its predictions against inputs purposefully engineered to fool or otherwise tamper with the model. Protecting the training data means preventing users from using the model to discover characteristics about the data. It also means preventing any data, whether with respect to the model or the training set, from being improperly leaked by those who develop and/or maintain the system.

The user of the system seeks to protect the data that is sent to and processed by the system. When a user requests an inference over private data from an AI-enabled system, they need to be assured that their personal data will not be exposed – and in many cases this means not having the data exposed even to the owner of the system. AI-enabled systems are often capable of learning the characteristics and behaviors of users over time, thus it is important that users be assured that their data will not be used for purposes they do not intend or desire.

PETs are effective tools for defending against system owner and system user vulnerabilities. Different PETs address different attacks from the different adversary types, and multiple PETs can be combined to either address multiple vulnerabilities or more robustly address a single vulnerability.

3.1. Trusted Execution Environments (TEEs)

A TEE is an isolated compute space, separate from its host instance, that relies on hardware-based encryption to protect both data in memory and application-level code. Entities outside a TEE, including the host instance, are unable to see or alter the code or data used during execution. A program running inside a TEE is cryptographically attested to be the program that is intended to run. (Mé nétrey etal., 2022) Figure 6 shows how a TEE could be deployed to ensure that data sent to a model is neither seen nor tampered with during inference.

Using a TEE ensures that systems are protected from external compute providers, which is of great concern to system owners. Using a TEE also provides guarantees to system users, ensuring that their data is not logged or stored in ways outside of its intended use.

3.2. Fully hom*omorphic Encryption (FHE)

FHE allows users to perform computation on encrypted data and models; that is, data in use is never decrypted. However, in its current-day state, it is very time- and space-intensive, so it must be implemented very conscientiously. Within the context of AI, a realistic use case for FHE is performing secure AI inference over smaller models in cases where increased inference latency is permissible. For example, DARPA is currently working on a program called Data Protection in Virtual Environments (DPRIVE) which aims to run inference over a 7-layer convolutional neural network performing inference against CIFAR-10 dataset in under 25 ms. (DARPA, 2020) Figure 7 shows how FHE could be used to privatize an external, inference process.

FHE allows encrypted interaction with an AI model and data sharing without risking misuse. There is strong potential that the time and space requirements of FHE will be improved through algorithmic (Lee etal., 2022), software, (Badawi etal., 2022; Kim etal., 2022)and hardware (Cammarota, 2023) developments. It is expected that upcoming developments in FHE will result in speed-ups of at least five orders of magnitude (Cammarota, 2022), and we expect to see further improvements as the technology continues to mature.

3.3. Federated Learning (FL)

Federated learning (FL) uses multiple clients’ data to build a shared AI model while keeping each client’s training data local; no client can access any other client’s data. Each client trains a model locally then sends that model (along with the number of examples that were used to train the model) to the server. The server then performs a weighted aggregation of all the models and sends the aggregated model to each client in the federation. This architecture relies on edge nodes with enough compute capacity to update their local model and may risk data de-anonymization (Gong etal., 2023; Yang etal., 2023) if not used in tandem with another PET. Figure 8 shows an example of how FL could be used in a centralized system to update client models without sharing client data.

FL directly addresses the concern of malicious insiders within the context of a collaborative group. Normally, distinct parties would be very hesitant to work together on AI models, because then all parties’ data needs to be shared. One party could reveal the sensitive contents of others’ data, even if that party had previously been deemed an ”insider.” FL greatly reduces the amount of trust required to collaborate on AI models.

With FL, the collaborators typically use a third-party server to perform the aggregation of their models. They can leverage an outsider for this because only model weights are sent to the server, not any data. However, to prevent the server from reverse-engineering these model weights, another PET such as FHE can be incorporated.

3.4. Summary of PETs Selection

The three PETs in this section have been selected for both their high utility and their wide applicability to a range of AI-related use cases. TEEs and FHE are both especially useful for securing the inference process. (Narra etal., 2019; d’Aliberti etal., 2023) With them, an organization can make an AI model available to a group people without ever seeing the data those people send to the model. FL is particularly useful for the training process; data from many parties can be leveraged without being revealed. TEEs and FHE can also be used for the training process, although the time and space demands of FHE in particular make training very difficult to scale. (Sidorov etal., 2022) The use of TEEs for training is more feasible than the use of FHE, but (1) it does not provide the same distributed computing benefit that FL does and (2) current, leading implementations of TEEs only isolate CPU and RAM (that is, they cannot leverage GPUs, and they do not have non-RAM storage, so they have very high memory requirements). (Services, nd; Azure, 2022)

These three PETs were thus selected because together they cover the main phases of the AI-enabled systems lifecycle, development/training and deployment/inference. There are, however, other PETs that can also be useful to AI-enabled systems:

These PETs were not described in detail because they pose more serious tradeoffs compared to TEEs, FHE, and FL, particularly in regards to model performance. They are, however, still worth consideration if implemented conscientiously.

3.5. PETs for Model Integrity

The above PETs address the confidentiality of data and models, but they do not address the integrity of model behavior. In the same way that current-generation cryptographic methods are used to ensure the integrity of data at rest and in transit (e.g., hashing), next-generation PET protections ought to address the integrity of AI models. The goal of hashing is to validate that data will accomplish its intended purpose, and it is important to carry this parallel into AI-related data-in-use scenarios.

As mentioned in section 2.2, adversarial attacks are a great example of a threat to the integrity of model behavior. Existing methods for addressing this threat include randomizing inference input data, (Xie etal., 2018), purposefully including adversarial examples in training data, (Bai etal., 2021) regularizing models to prevent input data sensitivity, (Jakubovitz and Giryes, 2019) or simply attempting to detect adversarial inputs directly. (Ahmadi etal., 2021) However, none of these methods provide formal guarantees that the model will behave as expected, and such guarantees are important for high risk models such as those used by autonomous vehicles. This, then, is an important gap that future PETs could help fill.

4.1. Use Case Applicability

The first step of integrating a PET into an AI-enabled system is to clearly understand the system’s use case and formally define the utility that the PET would provide. The technical objective, the threats to the system, and all relevant privacy and security requirements should be documented. AI system architects with experience using PETs will be able to select an appropriate PET (or set of PETs) to protect against data-in-use vulnerabilities. To leverage this experience, many organizations are collaborating to collect, curate, and publish use case repositories for open consumption. (ENISA, 2019) Consider the following questions to guide a use case applicability assessment:

4.2. System Impact

PETs provide innovative protections, but they may also require performance tradeoffs. A clear set of standards for PET performance characteristics is still under development, although significant progress has been made in the past five years. (Albrecht etal., 2018; ISO, 2023b) These up-and-coming standards aim to define cryptographic schemes and security parameters. Furthermore, standards for security and privacy best practices for AI-enabled systems in general are actively being developed. (ISO, 2023a) A key step toward implementing PETs in these systems is understanding new standards and how they relate to existing system requirements.

From an integration perspective, one must consider how the introduction of PETs affects the current system security level. PETs are still in their relative infancy, and new attacks against them will likely be developed in the future. PETs must be integrated in a way that allows for cryptographic agility; otherwise, they may have a net-negative impact on system security in the long run. However, with thoughtful design and a flexible integration plan, PETs have the potential to greatly enhance overall system security.

Currently, privacy and performance are typically at odds with each other; enhanced privacy can come at the cost of downgraded performance, and vice versa. PETs have the potential to undermine intended system functionality, and it is important to consider that tradeoff during evaluation. Consider the U.S. Census Bureau’s effort to minimize disclosure risk via differential privacy (DP). (Bureau etal., 2023) Though DP is effective at protecting the identities of individuals represented in a dataset, the “noise” that it injects into data necessarily perturbs it and can reduce statistical accuracy. The Census Bureau determines voting districts and allocates federal funding based on population statistics. Small perturbations in census data have the potential to produce outsized impacts on real-world decisions. Thus, in this case, statistical safeguards that protect the fidelity of data must be put in place to ensure the appropriate balance of privacy and accuracy. Another example might be a healthcare use case in which a doctor needs immediate access to records from another hospital as part of patient care. If the use of a PET introduces increased latency into a system, it could be detrimental to patient outcomes. Thus, it is critically important to be mindful of how the performance characteristics of a PET effect a given use case. The following questions are helpful in evaluating PET-related tradeoffs.

4.3. Implementation Readiness

Implementation readiness concerns both the readiness of the technology itself and the readiness of an organization to adopt the technology. PETs are currently in a transitional state as they go from being prototyped in a lab to being deployed in the real world. Organizations pushing PETs toward deployment include large companies—such as Intel Corporation, a leader in TEEs (Intel, nda), specialized FHE hardware (Cammarota, 2023), and remote attestation (Intel, ndb)—and small companies—such as Duality (Rohloff, 2017), a vendor with mature service offerings based on contributions to open-source FHE. (Rohloff, 2022) While typical measures for software acceptance (e.g., static and dynamic code analysis, fuzzing, penetration testing) are important for guaranteeing the security of such systems, their cryptographic nature also requires formal guarantees of their functional characteristics. Organizations adopting PETs should implement formal verification methods to ensure that the system operates as expected. To aid in such efforts, some organizations are publicly publishing PET maturity assessments to reduce the level of effort required to fully assess a PET’s readiness level. (ENISA, 2019) An organization must also carefully consider its own readiness for adopting PETs. Though PETs focus on data security and privacy, they have wide-ranging implications for enterprise data architecture, data governance policies, potentially multinational legal requirements, and ethical matters (which greatly affect customer confidence). Moreover, an organization must consider standard technical matters, such as staffing needs (at the development, integration, deployment, and maintenance levels), security analysis and accreditation, and technological strategy around data and AI/ML. These two groups of factors, organizational and technical, must be evaluated jointly to ensure the most effective adoption of PETs. The following questions could help guide a discussion about adoption readiness: