Running the HTML5 version of rs “optimally” is a bit hard to determine at the moment. (I assume you mean html5 by saying rs3).
You can certainly upgrade your existing computer by buying more ram (assuming your computer can support it) and buying a decent graphics card, likely ~100-200$ just for a good one, though there are some good sub or at 100 ones.
But’s not really known yet if that alone is going to make html5 rs work any better for you. Jagex has claimed that players are likely going to need an i5 processor or higher and GTX 200 (or equivalent) or higher to get the mid/high settings to run on html5…and that is when it’s fully optimized.
The real question is going to come down to if your CPU can handle it and that atm is a big unknown, given the above. The performance ability between yours and a i5 can be pretty noticeable so even after optimization, you still might not be able to play html5 on your current or even slightly upgraded computer. (html5 is in of itself cpu heavy)
If you are only planing to upgrade because of html5, I would hold off until the beta is far more optimized, currently my computer has a hard time getting frame rates above 20fps in the beta and my setup is rather high end. Holding off will give you a more realistic view of what you will/may need.
IF you plan to make a new gaming computer for things aside from RS Gold(which isn’t too hard, but might be confusing for those who never did it) then you are likely looking atleast 5/600.
i5 3570k **200 (ofc one can always go with a cheaper i5)
then ofc there is ram, power supply unit, motherboard, case, hdds)
If you think you are interested in building your own computer, should check around for guides to give you an idea of what to expect.
Some advice from someone who has built tons of custom rigs (some of them liquid-cooled) for self, friends, & school, from someone who has played RS for almost 10 years, and someone who has beta tested HTML5 from the beginning (main account: Victor Simek):
Hold off on updating your system for now. Right now the beta’s performance is all over the map for different users. Even the JMods have admitted that. Some people are getting good performance, others terrible performance, and it doesn’t seem to have a lot to do with specs. Even if you went for the absolute top-of-the-line now, there’s no guarantee that you’d get great performance. I have a rig that has an i7 @ 3.07Ghz, 2 GTX 560s in SLI, and 3 SSDs in RAID0 getting over 600Mb\s read and my performance in RS is by no means higher than anyone else’s at this point.
I would wait until the beta is more mature. Wait until the beta’s performance is more consistent with the specs of the computer that’s running it. Then you’ll have an accurate picture of what kind of system to go for, and you’ll avoid waisting money. Who knows, once the beta is fully optimized\mature, you may not even need to upgrade; your current computer may be enough.