In Maya there is no switching between openGL and DX, as its cross platform openGL is the only option (a d3d.dll is required at install for HLSL support and you can write your own DX viewport, but lets assume things are vanilla).
MentalRay uses RAM a bit different from the software renderer IIRC, how far into the render do you get before the crash?
Might be worth having Task Manager open when you start the render and see what memory usage is occurring.
There are a lot of particles in the scene, are these all cached out before you start rendering.
Also mentalRay will open a commandPort when it starts, some firewall applications have an issue with this. Could cause things to trip up, a long shot but hey hoo.
Hope you get it figured out see we can see the result
There is an exception to every rule, apart from this one.