Bakersfield, Calif., became more diverse and culturally expansive while the former House speaker was increasingly caught up in Washington’s barbed politics.