Not knowing how the ImageHandler.ashx works, but it's not an easy task to resize palette based and transparent images. You could use the popular framework ImageResizer.Net. You can also search StackOverflow for some answers http://stackoverflow.com/questions/30569/resize-transparent-images-using-c-sharp.
I think you should be able to change a setting called imgExtension in Web.config under episerver.community --> sites --> site --> imageGallery. If you change this to ".png" it should hopefully work better. I remember having the same problem a couple of years ago and I think this solved our problem.
Johan, ImageHandler.ashx functionality is out of the box for Community. Community costs about 200000NOK. Why should I have to rewrite or replace default image handler? It points out that Community is broken by desing. There is no value for money.
Martin, I allready changed format from .jpg to .png but it does not work.
Community gallery's transparent background images are displayed with black background when image is resized (from ImageHandler.ashx). When image is displayed in its real resolution without resizing everything is ok. How can I handle transaparent image resize?