Victor Olaiya was born on 31 December 1930, in Calabar, Cross River State, the 20th child of a family of 24. His parents, Alfred Omolona Olaiya and Bathsheba Owolabi Motajo, came from Ijesha-Ishu in Ekiti State. Olaiya came from a very rich family. His father’s house called Ilọijọs Bar stood on 2 Bamgbose Street, Lagos Island, until it was demolished on 11 September 2016. At an early age he learned to play the Bombardon and the French Horn. After leaving school he moved to Lagos, where he passed the school certificate examination in 1951 and was accepted by Howard University, US, to study civil engineering. Olaiya instead pursued a career as a musician, to the disapproval of his parents. He played with the Sammy Akpabot Band, was leader and trumpeter for the Old Lagos City Orchestra and joined the Bobby Benson Jam Session Orchestra.
In 1954 Olaiya formed his own band, the Cool Cats, playing popular highlife music. His band was chosen to play at the state ball when Queen Elizabeth II of the United Kingdom visited Nigeria in 1956, and later to play at the state balls when Nigeria became independent in 1960 and when Nigeria became a republic in 1963. On the latter occasion, Olaiya shared the stage with the American jazz musician Louis Armstrong. During the Nigerian Civil War of 1967–70, Olaiya was given the rank of a lieutenant colonel (honorary) in the Nigerian army and his band played for the troops at various locations. The Cool Cats later travelled to the Congo to perform for United Nations troops.
Olaiya renamed his band to the All Stars Band when they played the 1963 International Jazz Festival in Czechoslovakia.