English is messy and often words have more than one meaning. Because of its history the word fast has three meanings in Modern English (1. quickly 2. constant and secure 3. not eating voluntarily). We also have many metaphoric phrases created from these three meanings.
Some of these phrases using fast (as in constant and secure) are from the depths of time. From the Norse we get the expression fast by meaning near or close by. Fast and loose was a mediaeval English cheating game, but the term to play fast and loose became a reference to avoiding or ignoring rules.
Colour can be fast. A young man I knew once washed all his white shirts together with his red socks. The socks were not colour fast. So the colour from the socks stained all his shirts. He was left with no choice, but to wear pink, in doing so earned his nickname Pinky. The term colour fast has been around for more than 300 years. (To be acid fast is a similar concept to colour fast. It means that acid will not wash away colour and this property was used to identify certain bacterial diseases in the late 19th century. ) A fastness (from Old English) is a place that is difficult to attack, either a strong castle or today a mountain that is difficult to climb. However the same word has been used in Modern English to refer to the property of being colour fast.
Some other fast (as in quickly) metaphoric expression are modern, being less than 300 years old. Clocks are not ancient, but referring to a clock or watch as fast is more than 200 years old. Young males hope to find fast women who will join them in bed quickly. A person who indulges in fast living takes risks, enjoys vices to excess and dies young.
Many metaphoric expressions using the word fast (as in quickly) were created in the 20th century. To fast talk someone or to pull a fast one is to trick them. A fast buck means easily and quickly earned money. Fast food may be purchased and eaten quickly, but is of questionable nutritional value. On a highway, people may drive in the fast lane. The term fast track was originally used in horse-racing, but later gained the figurative sense of rapid promotion at a company. People used to fast-forward audiotape, but now the word is used for all types of recordings.
Usually the word fast means quickly, but be prepared for the older meaning. It is used in many common metaphoric phrases. Some of these can be guessed, but most should be avoided when talking to a second language learner or writing formal documents, such as a business letter or contract.