Fruits exhibit a vast array of different 3D shapes, from simple spheres and cylinders to more complex curved forms; however, the mechanism by which growth is oriented and coordinated to generate this diversity of forms is unclear. Here, we compare the growth patterns and orientations for two very different fruit shapes in the Brassicaceae: the heart-shaped Capsella rubella silicle and the near-cylindrical Arabidopsis thaliana silique. We show, through a combination of clonal and morphological analyses, that the different shapes involve different patterns of anisotropic growth during three phases. These experimental data can be accounted for by a tissue-level model in which specified growth rates vary in space and time and are oriented by a proximodistal polarity field. The resulting tissue conflicts lead to deformation of the tissue as it grows. The model allows us to identify tissue-specific and temporally specific activities required to obtain the individual shapes. One such activity may be provided by the valve-identity gene FRUITFULL, which we show through comparative mutant analysis to modulate fruit shape during post-fertilisation growth of both species. Simple modulations of the model presented here can also broadly account for the variety of shapes in other Brassicaceae species, thus providing a simplified framework for fruit development and shape diversity.