Why? If the training weights are created and distributed in violation of copyright laws, it seems appropriate to punish those illegal training weights.

Because they aren’t illegal and they don’t violate copyright. People keep wanting them to be against copyright, but that’s just not how copyright works. There either needs to be amendments to copyright law in order to cover this case, but those changes would need to be very carefully tailored. It would be way too easy to make something that’s either overly broad and applies to a bunch of situations it wasn’t intended to, or way too narrow allowing for easy circumventing.

In fact, all that people really are asking for, is for a new set of training weights to be developed but with appropriate copyright controls. IE: With express permission from the artists and/or entities who made the work.

While that might appease some people, it wouldn’t appease everyone. There are a lot of workers in the creative fields that are feeling incredibly threatened by generative AI right now. Some of these fears are certainly overblown, but it’s also true corporations are going to be as shitty as possible and so some regulation is probably in order. That said, once again, copyright just doesn’t seem to be the right tool for the job here.