Library Design#
NumPy vs. JAX#
While this library centers around the JAX ecosystem, it heavily utilizes standard Python and NumPy for specific modules depending on their computational requirements.
When we use JAX#
JAX is used for the core training logic that runs on hardware accelerators
(TPUs/GPUs) and benefits from XLA compilation (jax.jit()), automatic
differentiation (jax.grad()), and vectorization (jax.vmap()).
Gradient Clipping: As a key part of any training loop, gradient clipping is implemented in terms of JAX, utilizing its
jax.grad()andjax.vmap()transforms.Noise Addition: By default, noise addition is implemented in terms of JAX and is often implemented directly in the training loop using JAX-native pseudo-random number generators. It is possible to also do noise addition host-side (in NumPy) with a wider variety of random number generators, which can be helpful in situations where cryptographic security is necessary.
Matrix Factorization (
jax_privacy.matrix_factorization): Finding the optimal matrix factors requires solving a numerical optimization problem with gradient-based solvers. We use JAX to automatically compute gradients of loss functions, and JIT-compilation provides a significant benefit for some strategies. Moreover, some of our parameterizations benefit significantly from running on accelerators (likejax_privacy.matrix_factorization.dense).
When we use NumPy#
NumPy is used for host-side (CPU) operations that are not part of a typical training loop. This code often involves computations that are not highly compatible with JAX’s computation model, like dynamic loops, or do not benefit much from JAX, like single-use functions.
Privacy Accounting (
jax_privacy.accounting): Computing privacy guarantees and calibrating hyper-parameters often involves numerical integration or precise combinatorics. Because these are evaluated offline or infrequently, there is no need to run them on accelerators, and XLA compilation would not provide significant benefit.Batch Selection (
jax_privacy.batch_selection): Data loading and batching often occurs on CPUs, before being fed to the accelerator in the train step. As such, our batch selection API is defined with respect to NumPy and runs on the CPU as well. Users can pass in a custom NumPyGeneratorto control the behavior of the (pseudo)-random number generator.Auditing (
jax_privacy.auditing): Empirical privacy auditing calculates privacy bounds based on canary score distributions. Like accounting, it heavily uses statistical functions (scipy.stats), numerical optimization (scipy.optimize), and offline computations that don’t need accelerator performance.