The paper offers an overview of the work carried out by the Yemeni National Dialogue Conference (2013-2014), and which proposed the creation of a federal state with more open inclusive political order to soothe decades of internal tensions, north (Huthi) and south (separatist movement). It offers a “law and/in society” perspective on earlier attempts to propose decentralized mechanisms of governance in contemporary Yemeni history. Promises to anchor local structures of governance keep recurring at critical junctures in Yemeni history (in 1963 and 1964, when the Zaydi Imamate was toppled by republican revolutionaries; in 1970 during the finalization of the YAR "permanent constitution”; during debates in the mid-1960s prior to the establishment of South Yemen (PRDY); at the Amman agreement in Febr. 1994 before the civil war; and in the 2011 revolution). This begs the question of how path-dependent these discussions for decentralized forms of authority in contemporary Yemen may be and how other legal institutions favor, or not, the emergence of a federal order in Yemen. In a country characterized by an already or de facto decentralized form of governance (with state-sanctioned legal norms coexisting next to customary and Islamic laws—qadi justice), one can wonder why formal (political) decentralization efforts always fail. The paper offers an overview of how local structures, tribal, and customary law (‘urf), and the practice of embodied and relational Islamic justice (via qadi) have kept influencing and mediating social antagonism in a context where the central state has remained far from fully consolidated. The paper, based mostly on the secondary literature, with a dozen interviews with Yemeni actors involved in the National Dialogue, analyzes how informal mobilization within civil society, and past institutions (such as the makhaleef, old-time Islamic territorial units presented as a solution for the drawing of federal provinces in 2014) can provide insights and solutions to a faulty state-construction. Yemen epitomizes thus some of the “classical” dilemma of legal pluralism and tensions between a patriarchal order and legal-rational forms of authority.