Built in 1962, Swartz Gym is home to Ferrum's men's and women's basketball, volleyball and men's wrestling teams. The facility includes offices, a satellite sports medicine office, storage, locker rooms and four classrooms.
Swartz Gym, which seats 800 fans, has hosted NCAA Division III women's basketball first round games twice and a men's first round game once. The facility is also used by the college as an indoor commencement venue as well as for various non-athletic functions.
The Swartz Gym Pool is located on the lower level, and is home to the Panther men's and women's swimming teams. Above the gym is a golf practice area, complete with a simulator.

Ferrum's golf simulator is located on the third floor.