Can I use any engine? Or only visual novel specific? (for example I want to use godot)

You can use any engine, but you need submit a visual novel.

3D, RPG, etc is not allowed

I plan to use AppGameKit (AGK) which should be up to the task.  It is not specifically a visual novel engine, but it is a development platform which is versatile enough to be used for that purpose.