The paper argues that federalism, as proposed in the Yemeni draft constitution of January 2015, should not automatically be seen as an alien concept. If federalism is seen as a new form of decentralization, then, one can identify a kernel of widespread commitment and expectation for forms of decentralization in Yemeni republican life. After a brief overview of earlier attempts in Yemeni republican history to create forms of decentralization, the paper concentrates on the work carried out by the Yemeni National Dialogue Conference (2013-2014), and which proposed the creation of a federal state with more open inclusive political order to soothe decades of internal tensions, north (Huthi) and south (separatist movement). Pros and cons of a federalist Yemen are discussed in relation to public Yemeni debates on the matter and with interviews with stakeholders.