Yes, exactly, it seems like we app developers don't really have any control over this. The Dalvik VM is trying its best to accomodate all the apps running on a given device, but it's not perfect.
Lately I've been getting bug reports for my app that seem to be caused by mods like apps2sd and CyanogenMod. Don't get me wrong-- these are amazing pieces of software developed by amazing programmers. But it kinda sucks to have their side effects leak into our apps in the form of force closes, slowness, and lower Market ratings.
At least it's the minority of users that are reporting these problematic side effects.