In 1992 Bailey became the Suffragan Bishop of Dunwich in the Diocese of St Edmundsbury and Ipswich. During that time he was chairman of the finance committee of the Church’s Advisory Board of Ministry, dealing with the funding of ordination training.
His stay in Suffolk was brief, and in 1995 he became Bishop of Derby. The diocese, as well as containing rural areas, included considerable industrial activity and a large immigrant community. He made an immediate impact and one part of his programme was to visit pubs and nightclubs in the city centre. He helped to establish the Derby multifaith centre and was a governor of Derby University, which awarded him an honorary doctorate.
Bailey was known as a pastoral and caring bishop who remembered people’s names and mixed with all ages and faiths. On the national scene he was for three years the chairman of the Churches Main Committee, which acts as a liaison group between all the churches and government departments. During his time in Derby, he was also Clerk to the Closet, a role that involves advising the Queen about the appointment of her Honorary Chaplains. For this work he was appointed a KCVO. He was a member of the House of Lords from 1999 to 2005. He retired to Gloucestershire, where he was an assistant bishop and served on the Gloucester Police Authority.