Eventhough the pair gained early success, it lasted less than a year due to the arrest of Mulligan on drug charges. Due to this dissolution, Baker forms his own quartet with Russ Freemen, Carson Smith, Joe Mondragon, jimmy Bond, Shelly Manne, Larry Bunker and Bob Neel. Once again Bakers team had tremendous early success through the live performances and also through the albums produces in between 1953 and 1956. Due to this success, he was names Metronome magazine's top trumpeter in 1953. He achieved this beating the era's best trumpeters Miles Davis and Clifford Brown. This was a significant milestone in his career. He was also gifted with a delicate voice which later helped in producing some high quality vocals such as "Let's get lost" and many more. In addition he also enhanced and grabbed international attention through European tours in 1955 and in 1956. His movie star looks were tackled by the Hollywood movie makers in 1955, where he made his debut acting in the movie Hell's horizon. He then became an idol of the West Coast "cool school" of jazz. Even though he was in the run for fame, he was stuck by the evil of drugs.
During 1950s, Baker started using heroin which resulted to be his greatest lifetime addiction. He lost his valuables including his instruments to buy drugs, he was put into prison for drug-related offenses in Italy. He was deported from West Germany to the US because of drugs. He settled in Milpitas, Northern California. He was also assaulted while trying to buy drugs at a restaurant and hence his embouchure was ruined. After developing a new embouchure resulting from dentures he got back to Jazz and started performing again pursuing his career. He partnered with guitarist Jim Hall. After sometime he went back to Europe and started performing and recording and was also assisted by one of his friends namely Diane Vavra. From 1978 his life became more prosperous and brought about a lot of fame.