My guess...
From a graphics standpoint
-------------------------------
There's download lag: The amount of resources you have to download to see all of the graphics (heads, bodies, shields, swords, hats, ganis, tilesets, levels, NPCs). The least of these would be ganis, tilesets, levels and NPCs because they're fairly persistent and can be cached. Heads, bodies, shields, swords and hats change depending on who's in the level (this goes with text rendering and emoticons as well). In my dev days the gmap loaded the eight levels surrounding where you were ahead of time, which helps with download lag when going between levels.
There's rendering lag: More people moving about means more to render. Any graphics that are animations add to this.
From a server standpoint
-----------------------------
The more actions sent to the server, the more processing it has to handle. The server is a huge bottleneck of activity.
-----------------------------
Most of it could probably be reduced by upgrading your CPU and RAM, but is pretty dependent on congestion (how many players are online). Server lag is pretty dependent on internet connection and also congestion.