Men with inflammatory bowel disease have four to five times higher risk of prostate cancer, according to a new 20-year study.