If you have a super heavy robots.txt file beyond 500KB, then you must limit it to a reasonable size as it can cause serious issues with your site’s health in the Google results.
Google’s John Mueller on his Google+ page reminded webmasters that Google has a limit of only being able to process up to 500KB of robots.txt file. If you have a super heavy robots.txt file, and it is beyond 500KB, GoogleBot can get confused.




