Yes with Activitypub federation you generally have to trust server admins with the data they handle (similar with private messages). Though I suppose it would be possible for communities to anonymize votes when announcing them to followers, then only the admin of the community’s instance could see who voted.