Microbial eukaryotes present a stunning diversity of endomembrane organization. From specialized secretory organelles such as the rhoptries and micronemes of apicomplexans, to peroxisome-derived metabolic compartments such as the glycosomes of kinetoplastids, different microbial taxa have explored different solutions to the compartmentalization and processing of cargo. The basic secretory and endocytic system, comprising the ER, Golgi, endosomes, and plasma membrane, as well as diverse taxon-specific specialized endomembrane organelles, are coupled by a complex network of cargo transport via vesicle traffic. It is tempting to connect form to function, ascribing biochemical roles to each compartment and vesicle of such a system. Here we argue that traffic systems of high complexity could arise through non-adaptive mechanisms via purely physical constraints, and subsequently be exapted for various taxon-specific functions. Our argument is based on a Boolean mathematical model of vesicle traffic: we specify rules of how compartments exchange vesicles; these rules then generate hypothetical cells with different types of endomembrane organization. Though one could imagine a large number of hypothetical vesicle traffic systems, very few of these are consistent with molecular interactions. Such molecular constraints are the bottleneck of a metaphorical hourglass, and the rules that make it through the bottleneck are expected to generate cells with many special properties. Sampling at random from among such rules represents an evolutionary null hypothesis: any properties of the resulting cells must be non-adaptive. We show by example that vesicle traffic systems generated in this random manner are reminiscent of the complex trafficking apparatus of real cells.