That is not true. For example, you must have a pass to park at the trailhead for the South Fork trail. There is nowhere to park there without a pass. If there is pavement and a toilet, you need a pass.
It’s actually pretty interesting, how it came to that. It used to be that you needed a pass at every trailhead. That hasn’t been the case for almost a decade. There were multiple federal court court cases that led to the current fee area situation. Search for “Mt Lemmon fee court” and you’ll find links to Adams v. USFS and US v. Wallace, which is a start down a pretty interesting rabbit hole.
If you’re parking at a San Gorgonio wilderness trailhead, you have to have a federal lands fee area pass of some sort displayed at the south fork trail, no matter what, and at the Vivian Creek trail, if you’re parked on the pavement. That’s it, in a nutshell. You don’t need a pass at any other trailhead.
You do need a hiking permit tho, if you are planning to actually enter the wildness, which is generally any place above 7,200 feet.