The 26th and last letter of the Modern English alphabet, is comparatively new to English1. The Modern English alphabet is from the Latin. The Latin letter Z was originally only used in Old English for foreign words. In Old English the letter S was used for the S sound (/s/), the SH sound (/ʃ/), and the Z sound (/z/). Over the centuries many words with a Z were borrowed by English and spelt with a letter Z.
This association of the letter Z with foreignness was so strong that many English words of French origin used Z when spelt in English to show that they were foreign. This change happened even if the original French spelling did not have a letter Z; the English added one. The word hazard is from the Old French hasard. The word magazine is from the Middle French magasin (originally meaning warehouse). The word seize is from the Old French seisir. The word size is from the Old French sise.
There are some exceptions to this pattern. The word dozen came from French with a z-spelling. Another exception is the word freeze, which is from the Old English freosan. Someone thought that the Z-sound should be spelt Z and made a spelling mistake, which never got corrected. American spellings sometimes do the same thing.
There are some other strange reasons why the letter Z gets used. Middle English had the letter ȝ (yogh). The invading Normans did not like non-Latin letters and often replaced it with other letters. Early printers brought their presses from Germany and there was no letter ȝ in their letter sets. So they choose the letters Z, GH or Y as substitutes to replace the old ȝ . For example the name MacKenȝie became MacKenzie2.
The letter Z is so uncommonly used that it is worth 10 points in English Scrabble sets. As the Earl of Kent said in, Shakespeare's King Lear, "Thou whoreson Zed, thou vnnecessary letter." This letter Z is still unusual and often a mark of the foreignness of a word.
1. It has been a letter for a long time. The letter Z came from the Old Semitic proto-alphabets, into ancient Hebrew, into the Phoenician alphabet, then into Greek, and finally into Etruscan and Latin.