Bitcoin uses a public-private key encryption system. The public key is your bitcoin address and the private key is what allows you to spend your bitcoins.
The private key is never shared with anyone and is what you use to sign a transaction. The signature is then verified by the network using the public key.
The elliptical curve digital signature algorithm (ECDSA) is what is used to generate the signature. ECDSA has been chosen because it is believed to provide good security and performance.