For software to be trustable the public needs be able to see what it does (not necessarily as an individual but collectively). Normally that means we need the source code but “AI” can mean a system like an artifical neural network which isn’t enough to understand what it is actully doing (even with the training data).