I’ve noticed the same slowness when in the dashboard, when using shared hosting. On dedicated or WordPress hosting, it’s faster.
I can confirm it’s the hosting… The dashboard is lightening fast using local test server (MAMP) which means the hosting server is the main issue, or other plugins.
You can install a plugin like WP Query Monitor to see for yourself, Swift is not the issue. https://wordpress.org/plugins/query-monitor/
I am in for the info on GD Image Library. Very curious.