The garden in the title is a reference to the lost paradise of Genesis but also the hortus mentis of the philosophers, as well as the hortus conclusus from the Song of Songs. To sum up: all the places of solitude, where the mind can recreate itself and perform its act of contemplation.
How vainly men themselves amaze,.
To win the palm, the oak or bays.
And their incessant labours see.
Crown'd from some single herb or tree.
Whose short and narrow-verged shade.
Does prudently their toils upbraid.
While all the flowers and trees do close.
To weave the garlands of repose.
The speaker places himself in a felicity which he earned by abandoning the busy companies of men, mocked for toiling in order to win a garland of recognition, woven from the leaves of either the palm of a triumphant general, the oak of the public leaders or the bays of an artist. They could have the whole tree without so much activity and nervousness. The reward for inactivity are the garlands of repose, as Marvell calls them. The tone is rather amused - the crown braided up - as a girl braids her hair. But upbraid as a reproach is also implied: fools indeed are all those hectics who prefer the scanty shade of the garland to the shade of the tree.
.
Fair Quiet, have I found thee here.
And Innocence, thy sister dear.
Mistaken long, I sought you then.
In busy companies of men.
You sacred plants, if here below.
Only among the plants will grow.
Society is all but rude.
To this delicious solitude.
Now the speaker moves from the physical garden to the hortus mentis - the garden of mind. Experiencing solitude in the garden helps you find yourself. What we also see is that the poem works by overturning the conventional views. The society is rude (this pun is not visible for people of the 2oth century, in former times society was associated with civilite. Once upon a time there was Innocence in the garden - but not solitude, we will return to it later. .
No white nor red was ever seen.