The Anglican parish of Kelso was founded in 1825. This was the first church in Australia to be consecrated by a Bishop and the first church built in inland Australia. Holy Trinity Church is built on a hill overlooking Bathurst and is surrounded by a historical cemetery.