Fertility is the factor that defines the growth rate of farmable tiles within your colony. The Fertility overlay can be activated by clicking the overlay menu at the top left of the UI and selecting "Fertility" from the drop down overlay menu.
Fertility does not change as farms are added and removed from your territories, as it is generated before the game begins (See World generation).
Farms created within tiles with 100% Fertility will grow its crops at 100% efficiency. The efficiency of a farm can be determined by the average fertility of all tiles selected for a single farm plot (Maximum number of farm tiles up to 64).