The consumer group Which? estimates that seven million debit and credit card holders are putting themselves at an increased risk of having their account emptied by criminals.
It says they either tell people their PIN, or even write it down.
A third of people who admitted they had written down their PIN said they kept it in their wallet or their handbag. 36% had it written down somewhere at home, while 9% had made a note of it in their office.
If your card is stolen, the banks may not pay compensation if they can prove you didn't keep your PIN number secure.
Phil Jones from Which? commented: "They're not going to look kindly on you if you have put the PIN, say, in your bag with your card because that could arguably be seen as careless behaviour. They may well turn down any claim you make and leave you out of pocket."
And of course it's worth remembering you can set your own PIN at a cash machine.