The metabolic requirements of hematopoietic stem cells (HSCs) change with their cell cycle activity. However, the underlying role of mitochondria remains ill-defined. Here we found that, after mitochondrial activation with replication, HSCs irreversibly remodel the mitochondrial network and that this network is not repaired after HSC re-entry into quiescence, contrary to hematopoietic progenitors. HSCs keep and accumulate dysfunctional mitochondria through asymmetric segregation during active division. Mechanistically, mitochondria aggregate and depolarize after stress because of loss of activity of the mitochondrial fission regulator Drp1 onto mitochondria. Genetic and pharmacological studies indicate that inactivation of Drp1 causes loss of HSC regenerative potential while maintaining HSC quiescence. Molecularly, HSCs carrying dysfunctional mitochondria can re-enter quiescence but fail to synchronize the transcriptional control of core cell cycle and metabolic components in subsequent division. Thus, loss of fidelity of mitochondrial morphology and segregation is one type of HSC divisional memory and drives HSC attrition.