I'm not sure exactly what you are doing but I could not get the scaleing thing to work with image view. You may want to take a look at this tutorial and its associated posts: novice-tutorials-f8/large-image-scrolling-using-low-level-touch-events-t11182.html
which starts out with image navigation but gets into zooming. The basic Idea in the code is to display this in two rectangles so you can navigate about the image. In zooming if the image becomes to small the rectange causes a stretch. To avoid this I started using the canvas.drawBitmap(bitmap, 0,0,null) as suggested by pksink when the image got too small.
if you want to use the image view I might recommend switching to center instead of center_crop and just ImageView.setImageBitmap(bitmap) the image you have directly to the image, it is alread scaled and should not be a problem.