Born in Swansea, Richards played local football for his schools and local club West End before joining the Cardiff City Academy. After two seasons, he was released and joined the academy at their South Wales rivals Swansea City at the age of fifteen. Richards initially started out as a midfielder but switched to playing as a defender under manager Roberto Martínez.
After spending two seasons with the academy and reserve sides, Richards made his professional debut for Swansea in a Championship match against Middlesbrough during a 3–0 loss on 15 August 2009. After Richards made 15 appearances during the 2009–10 season, he signed a new two-year deal with Swansea.
The start of the 2010–11 season saw Richards spend the opening six months as an unused substitute or in the reserve side. He did not make his first appearance of the season until 3 January 2011, during a 2–1 defeat against Leicester City following a defensive injury crisis where several first team players were ruled out. Throughout January and February, Richards was given a handful of first team opportunities. He made eight appearances during the 2010–11 season as Swansea went on to become the first Welsh team ever to gain promotion to the Premier League.
After appearing on the bench for several matches at the start of the following season, Richards made his Premier League debut when he came on as a substitute for Neil Taylor during a 3–1 loss to Norwich City on 15 October 2011. Twelve days later, he was rewarded with a new 18-month contract. He then made his first start for the club in the Premier League, playing in a 4–2 loss against Blackburn Rovers on 3 December 2011.
Loan to Crystal Palace in 2013
At the start of the 2012–13 season, Richards struggled to break into the first team over Rangel in the right-back position. Because of this, he stated his desire to leave the club on loan to gain experience and compete for his spot in the first team. However, he was forced to wait until the winter transfer window to secure a move. In the first half of the season, he made just four appearances for Swansea, all in cup competitions. At the end of January, he joined Crystal Palace on loan until the end of the season. He made his debut for the club five days later, on 30 January 2013 in a 1–0 loss against Huddersfield Town. He went on to make 11 appearances and appeared as an unused substitute in the 2013 Championship play-off final, as Palace beat Watford 1–0 to achieve promotion to the Premier League.
Loan to Huddersfield Town
After returning to Swansea, Richards made his first appearance of the 2013–14 season in the second leg of the Europa League qualifiers, a 0–0 draw against Malmö, helping Swansea advance 4–0 on aggregate. On 10 September 2013, Richards joined Championship side Huddersfield Town on a 93-day emergency loan. After not being able to obtain international clearance from FIFA in time to play against Doncaster Rovers on 14 September, he made his Town debut as a substitute in a 2–1 win over Charlton Athletic on 17 September 2013. His loan spell ended on 17 December after just 9 appearances for the Terriers. Following his return from his loan spell, Richards did not appear in another match until 16 February 2014 when he made played in a 3–1 loss against Everton in the last-sixteen of the FA Cup.
Return to Swansea
In the 2014–15 season, Richards made his first appearance of the season in the first round of the League Cup, in a 1–0 win over Rotherham United on 26 August 2014. He then made his first Premier League appearance in two years when he came on as a substitute in the 82nd minute of a 0–0 draw against Sunderland on 27 September 2014.
On 24 January 2015, Richards joined Championship side Fulham on a one-month loan deal. He made his debut for the club on 31 January 2015, starting in a 2–1 loss against Blackburn Rovers. After making five appearances, Richards had his loan spell with Fulham extended until the end of the season Richards went on to make 14 appearances for the Cottagers, all of which were starts.
On 2 July 2015, Swansea accepted Fulham's bid for Richards to transfer on a permanent deal. He signed a three-year deal with the option of a fourth. Richards' made 22 appearances in his only season as a permanent player at the club. He established himself as a regular when he was fit. However, ahead of the 2016–17 season.
On 19 July 2016, Richards joined Championship club Cardiff City on a three-year-deal in a swap move which saw Scott Malone join Fulham. He made his debut in a goalless draw at Birmingham City. In his first season with Cardiff, he appeared in a career high 26 league matches.
At the start of the following season, he made only eleven appearances in all competitions during the season as Cardiff achieved promotion after finishing as runners-up.
Richards represented the Wales under-17 and under-19 sides. He was called up to the under-21 squad for the first time in March 2009 to face Luxembourg. He won ten or more caps at all three levels during his youth career.
On 27 May 2012, he gained his first full international cap for Wales in their 2–0 friendly loss to Mexico in New Jersey, replacing Neil Taylor for the last ten minutes of the match. Wales qualified for Euro 2016 and Richards was called up into the final 23-man squad. He played his only game in the tournament as a late substitute in the team's first match, a 2–1 victory over Slovakia. Wales went on to reach the semi-finals before losing to Portugal. As of October 2018, Richards has had 14 international caps with Wales, including seven appearances in the starting lineup. He played in four matches in Euro 2016 qualifying and three in 2018 World Cup qualifying.