Saturday, April 13, 2024
Google search engine
HomeUncategorizedNeural nets fail on formal languages because the solutions aren't loss optima

Neural nets fail on formal languages because the solutions aren’t loss optima

[Submitted on 15 Feb 2024]

Download PDF

Abstract:Neural networks offer good approximation to many tasks but consistently fail to reach perfect generalization, even when theoretical work shows that such perfect solutions can be expressed by certain architectures. Using the task of formal language learning, we focus on one simple formal language and show that the theoretically correct solution is in fact not an optimum of commonly used objectives — even with regularization techniques that according to common wisdom should lead to simple weights and good generalization (L1, L2) or other meta-heuristics (early-stopping, dropout). However, replacing standard targets with the Minimum Description Length objective (MDL) results in the correct solution being an optimum.

Submission history

From: Nur Lan [view email]

Thu, 15 Feb 2024 15:25:30 UTC (1,766 KB)

Read More



Please enter your comment!
Please enter your name here

- Advertisment -
Google search engine

Most Popular

Recent Comments