After a solid high school career Williams decided on the University of Maryland where he would go onto NCAA stardom. While at Maryland he was named 1979 ACC rookie of the year, lead the NCAA in rebounding twice, was named all ACC twice and was one of the members of the USA men’s national team. He was however never allowed to compete in the 1980 Moscow games due to the United States boycotting the games due to the Soviet Union’s invasion of Afghanistan the year prior.

After three years at Maryland Williams went pro. The New Jersey Nets selected Williams as the third overall pick in the 1981 draft behind Mark Aguirre and Isiah Thomas. Williams would make an immediate impact on the Nets who went from 24 wins the season before to 44 and made the 1982 NBA Playoffs. For helping his team turn around as well as averaging 15 points and 12 rebounds Williams was named 1982 NBA Rookie-of-the-Year.

His first three years in the NBA were extremely solid. He averaged at least 15 points an 12 rebound in all three seasons and was selected as an all-star in three of them. Williams helped the Nets make the playoffs in all three of those seasons and in 1984 helped them into the second round where they lost to the Milwaukee Bucks. It would be another 18 years until the Nets again advanced past the first round.