Pretraining of a Swiss Long Legal BERT Model

We will scrape legal text in German, French and Italian to pretrain a Swiss Long Legal BERT model capable of performing NLP tasks better in the Swiss legal domain.


  • Lead school Business School
  • Institute Institute for Public Sector Transformation
  • Research unit Digital Sustainability Lab
  • Funding organisation Others
  • Duration (planned) 15.12.2021 - 31.07.2022
  • Project management Prof. Dr. Matthias Stürmer
  • Head of project Joël Niklaus
  • Partner Schweizerisches Bundesgericht


We see a clear research gap that BERT models capable of handling long mul- tilingual text are currently underexplored (gap 1). Additionally, to the best of our knowledge, there is no multilingual legal BERT model available yet (gap 2). Tay et al. [2020b] present a benchmark for evaluating BERT-like models capable of handling long input and conclude preliminarily that BigBird [Zaheer et al., 2020] is the currently best performing variant.

Course of action

We thus propose to pretrain a BERT-like model (likely BigBird) on multi- lingual long text to fill the first research gap. To fill the second gap, we propose to further pretrain [Gururangan et al., 2020] this model on multilingual legal text.