Q:

What happens when you place a Ghost Ward beyond the Sightstone's limit?

The new Sightstone seems like a fantastic value,

Unique: Ghost Ward - Consumes a charge to place a  Ghost Ward. You may have a maximum of 2 wards from this item at once.

What happens when you put down a third ward while two are still on the map?
Does the same happen with the Ruby Sightstone?

A:

It deletes the oldest ghost ward out at the time and places your new one down. See below for examples

Here is two wards from the sight stone

You can see the other ward fading out after I placed the third

After getting the ruby sight stone I can place a third ward with the other two out

However trying to place a fourth ward causing the oldest to disappear still

