Troubleshooting

Troubleshooting#

If you don’t find a solution for your problem here, look for similar issues on our GitHub.

This documentation is in active development and we are still gathering most common pitfalls and issues that might arise during the use of our library. If you have one, you are very welcome to contribute!

Loss does not go down#

Check whether it goes down with non-DP training. If it does not go down for non-DP either, then try usual ML remediation techniques like changing learning rate, using different optimizers. Such a problem might often happen if you use 16-bit mixed precision because more error is accumulated. In this case, accumulating fewer gradients or adjusting precision might help (see the Mixed Precision Training guide).

If it happens only for DP training, then it is worth noting that DP training in general converges slower than usual training. In this case, try to do more iterations, increase batch size (both physical and effective; see Handling Variable Batch Sizes), or increase the learning rate. For a comprehensive overview of pitfalls that can affect DP convergence and utility, consult Common Pitfalls in DP Training.