Milsom began his career as a youth player at Fulham, which he joined since eleven, before he was promoted to the reserve team. During the 2005–06 season he played regularly for the reserves and also captained the side on several occasions. Milsom initially played in the left back position before switching into a centre midfielder. In early 2008, Milsom signed a contract extension, which will keep him until 2010. Milsom make his full debut as a late substitute for Seol Ki-hyeon in the League Cup against Burnley.
Milsom made his league debut for Fulham on 18 February 2009 in a 3–0 defeat at Old Trafford against Manchester United, coming on as a replacement for Danny Murphy. Unfortunately one week later, Milsom sustained a knee injury that ruled him out of the entire 2008–09 season. Then in March 2010, Milsom signed another contract extension, which would keep him at the club until 2011. While at Fulham, Milsom considered his reserve manager Billy McKinlay as his mentor, citing his biggest influence on his career.
After failing to break into the Fulham first team, and to gain some experience, he spent a month on loan at then Football League Two side Brentford in the second half of the 2007–08 season where he played six times. He returned to Fulham on 14 March 2008.
He again went out on loan, this time to Southend, making a further six appearances. On 29 December 2008, Milsom would return to Fulham.
In March 2010, Fulham reported that Milsom will be joining Finnish club Turun Palloseura on loan from 19 April to 30 August 2010. Milsom would make 18 appearances in all competitions.
Milsom made his debut for the club two days after signing for the club, in a 2–0 win over St Mirren and then scored his first goal in the return game on 6 April 2011, which Aberdeen would lose 3–2. Having been in the first team, making twelve appearances and his own desire to stay at Aberdeen, Milsom signed a new two-year contract with the Dons on 22 April. After Aberdeen finished in ninth place, Milsom stated that he felt Manager Craig Brown would have helped the club qualify for European competition had he joined at the start of the season.
In 2011–12 season, Milsom started his season in central midfield and then scored his first goal of the season, in a 2–1 win against Inverness CT on 19 August 2011. In the first half of a 2–1 loss against Rangers on 28 October 2011, Milsom was tackled by Steven Naismith, which resulted in both players coming off injured. Milsom felt that Naismith should have been sent-off, claiming that the high tackle could have caused him a serious injury. Milsom would miss the remainder of the 2011–12 season after undergoing a knee operation.
In 2012–13 season, Milsom made his return in pre-season and scored in a 12–1 win over VfB Alstatte. However, in late-August, Milsom, once again, sustained a serious injury that would keep him out for a long time. Milsom made a return on the bench ahead of a match against Dundee, which Aberdeen would win 3–0. Upon making his return, Milsom found himself in and out of the first team, blaming "indifferent form and a niggling injury."
At the end of the 2012–13 season, Milsom was among ten players to be released by the club upon expiry of his contract.
After being on the bench for the first two games of the season, Milsom made his debut for the club, where he came on in place of Michael O'Connor in the 85th minute, in a 2–1 win against Crawley Town on 17 August 2013. He then scored his first goal for the club in a 3–1 win over Sheffield United on 7 September 2013, where he "controlled smartly before firing a low drive from the edge of the box which beat the outstretched hand of Howard before smacking the inside of the post enroute to the net."
Having broken into the first team and receiving good reviews from pundits, Manager Steve Evans was keen for Milsom to stay at the club. Milsom explained his good performance because of his repayment to Evans for signing him. On 31 October 2013, Milsom signed an 18-month contract at the club, which will keep him until the summer of 2015. At Rotherham United, Milsom was more used in the first team as a defender in the left back and was often injured including a thigh strain. Milsom returned to the first team following a fitness boost. He made 27 appearances in the 2013–14 campaign, helping the club to reach the League One play-off final. He was on the substitute and came on for Joe Skarz in the 77th minutes at Wembley Stadium as Rotherham secured promotion with a penalty shoot-out victory against Leyton Orient.
Ahead of 2014–15 season, Milsom retained his squad number at number fifteen. However, Milsom sustained an injury and made his return from injury two months later. Milsom made his first appearance for the club in the Championship, where he came on as substitute in the second half, in a 3–0 loss against Reading.
On 26 November 2014, Milsom joined Bury on loan until the New Year along with Bristol City defender Adam El-Abd. Two days later, he made his Bury debut in a 2–0 loss against Dagenham & Redbridge, and played once more before returning to his parent club. After making seven more appearances for Rotherham, Milsom was one of five players released at the end of the season.
He was released by Notts County at the end of the 2017–18 season.
On 13 August 2018, following a successful trial period, Milsom joined fellow League Two side, Crawley Town on a short-term deal.
On 31 August 2018, following the departure of first-team manager, Harry Kewell, Milsom followed the Australian to Notts County on a five-month loan deal with a permanent agreement in place for the following January, just three months after leaving the Magpies. He made his first appearance against Forest Green on 1 September 2018 and scored his first goal since returning against Crewe a few weeks later. Milsom went on to make a total of 24 appearances whilst on loan.
On 14 January 2019, Milsom signed on a permanent basis.
He was released by Notts County at the end of the 2018–19 season.
- As of match played 21 December 2019