Sometime website loads slow on IE but same website when we load on Mozilla Firefox, Google Chrome it is fast. This happens if you are giving restricted access to users.
IE need to download some updates, connect to OCSP servers so if these connections are blocked IE will keep on trying to connect to update server, OCSP server and after so many tries it gives up and loads the website.
Check following screenshot which shows connection made when we gave full access to a system having issues:
Work around would be allow OCSP, ms-updates for the affected system.