You can't use R Markdown format like papaja developed as R format within Quarto. So it won't work to just do that in YAML. Only quarto format can be used after the format key, and it would require developing a new custom format for Quarto similar to papaja.
format
key, and it would require developing a new custom format for Quarto similar to papaja.