The Apps are Too Damn Big(tobykeller.tumblr.com) |
The Apps are Too Damn Big(tobykeller.tumblr.com) |
As it only takes one large app to disable the whole update all process, it'll be interesting to see how this impacts the overall update numbers.
Here's an off-the-top-of-my-head potential solution:
Update Xcode to sensibly handle file structures within a project. You know, the blue folders. We currently use blue folders for all app assets because, well, can you imagine maintaining 1600 files all in on directory?
The problem with blue folders today is that they needle you with pain. You have to remember to do clean builds (scripts during the build don't seem to work reliably) and, in general, be very aware of them. Once they are part of your process its not a big deal, but I think it's high time that Xcode come to this century and do something that most other development systems have been doing for, I don't know, decades?
Anyhow, once that is in place, you could have a directory structure such as this one at the app root:
App Resources
App Resources @2X
App Resources ~ipad
App Resources ~ipad @2X
{
"ipod 1": "App Resources",
"ipod 2": "App Resources",
"ipod 3": "App Resources",
"ipod 4": "App Resources @2X",
"iphone 1": "App Resources",
"iphone 2": "App Resources",
"iphone 3": "App Resources",
"iphone 4": "App Resources @2X",
"iphone 4S": "App Resources @2X",
"ipad 1": "App Resources ~ipad",
"ipad 2": "App Resources ~ipad",
"ipad 3": "App Resources ~ipad @2X"
}
+ (UIImage *) imageNamedAndKeyForPath:(NSString *)name key_for_path:(NSString *)key
Apple then, would have a mechanism to only deliver the appropriate resource files during an in-device installation. iTunes could download the whole set and also be intelligent during to-device installation. They'd use the JSON file to have the developer tell them which files to deliver.
Any files left outside the designated directories would be treated as they are today: Everything goes to every device.
Also, other methods that take a path, such as "imageWithContentsOfFile" ought to have a version that uses the aforementioned mechanism while --and this is important-- letting me specify relative paths from the device-specific folder. If I have something like this:
<device specific folder>
+ character
+ arms
+ legs
+ head
+ background
+ trees
+ rocks
+ clouds
On first inspection this sounds like a good solution. I haven't thought it through entirely. I'd be interested to hear of what holes the idea might have.
Something like this could have a striking effect on app bundle size, particularly on older devices. I don't think that it places an undue burden on developers either, if anything it makes things easier. Heck I'm sure Apple could even find a way to make it a marketing point: "Four times as many apps in the same space".
1) UIImage already handles choice based on asset name. Apple could use the same asset choice logic server-side for each device, without the developer having to corral his assets into directories. It just needs to be expanded to handle ~ipad and ~iphone (which UIImage does not natively, but if they make this a standard, they could.) The fallbacks would, of course, be most-specific to least-specific: scale and device before common. One downside here is that direct path access to an image will not be fixed, but they already do static analysis for message names, you can find calls to bundle methods and keep those assets lying around.
2) JSON?! We have property lists, they're way more Appley and have framework support through-and-through.
3) While this might provide the ability for a developer to choose which resources are used where, Apple is not in the business of giving developers (or users) choices.
In other words, I don't think the onus should be on developers. They should be free to do what they do best: innovate about the real problems, not the platform's solvable deficiencies.
The above listed can also be applied to all existing app bundles, were Apple feeling adventurous. Parent's method can not.
Geez, thanks.
UIImage's handling of assets based on file name is a horrible band-aid.
Also, one has to think workflow. And, to make things more interesting, exaggerate. Let's make it 5,000 files in one directory. That's a mess. With device-specific directories an artist could create a workflow leading to easy maintenance and creation of files. No need to engage in funky naming conventions. The per-device files are simply located where they need to be and a mechanism is provided to identify which path serves a particular device type.
Another problem with UIImage is that there is no way to communicate to Apple which files are to be delivered to, say, an iPod touch. So, an external mechanism is required in order to encode that information. Hence the proposal for an external JSON (or whatever) file.
Why JSON. I like it. Supported by iOS 5.0 as well. Easy to maintain and exist externally to Xcode. This also means that I can write scripts or automation tools on a PC or Mac to automate the creation of assets and their placement within the proper structure. The file used within the app to communicate the per-device directory structure would also could also serve external purposes that cannot be covered with plists. In our case, as an example, all of our assets are being produced on PC's, not Macs.
> Apple is not in the business of giving developers (or users) choices.
That's another matter. Although my proposal wasn't about providing choices as much as it is about a mechanism to only deliver the necessary files to a device. Only the developer can make decisions about which files are appropriate, not Apple.
Ultimately, I don't care what technology might be used in order to enable something like this so long as it works and is reasonably open. Right now I'd love to be able to deliver only 1X files to iPods and ~ipad@2X files to the new iPad rather than forcing a huge download of 1,600 files to every device or the current project on our table. That's just silly.
Also, keep in mind that localization can also exist in graphical assets. You could have images with built-in text that need to be switched in based on localization. So, an English version of the app uses this set and an Italian version another set of assets. This might include sound files.
Depending on how this is handled it can start to explode the app bundle in impressive ways. Start with 100 images. Now you need 400 to cover all device display configurations. Then you need to replicate this on a per-language basis. If you cover, say, five languages, you are up to 2,000 files potentially delivered to every single device regardless of applicability. I realize that this is a corner case where all images need to be localized. No need to point that out.
I have not done any internationalization yet, so I don't know how this part works. For some cases it might not be practical to release a separate app in different languages. For example, a kids app that is able to operate in English and Spanish and sold through the US app store.
Let's say that a future iDevice comes with a better audio engine. Maybe it can do 5.1 surround? You could extend the JSON structure to include a section to define where the 5.1 surround files would be for the new device. Older devices wouldn't have to waste storage and download audio files that they can never use.
Apple to save a penny, costed heavily for everyone else involved.
As an iDevice user I am glad that I chose to buy all my devices with the absolute max storage available. Everything I have has 64GB. So, we are good for a little bit.
As a developer I cringe at the situation. I am currently working on an app that has about 400 graphics files that will ship with the app. If you do it their way you have to produce each one of these files in four different resolutions. This means that the image directory is now huge. It is three times larger than a scenario where we would have app bundles customized for each device.
In looking at the problem we decided to do the following. All of our files are produced at what would be @2X resolution but are named without the @2X. We are going to ship the app with just 400 files at a single resolution instead of 1600 files at four different resolutions. In the end, if you do the math, your app bundle will be significantly smaller due to the significant reduction in size of the images directory.
As far as performance is concerned, the images work very well all they way back to iPod touch v2.0 devices. No issues there. So, that's the plan.
Maybe what Apple is pushing all of us towards is a scenario where we are forced to detect device type and capabilities and load resources accordingly on app first run. Think about this for a moment. Now their servers don't have to push out so much content and you will have to foot the bill for your free graphics-intensive app getting three million downloads. I could be off base here. But, I just don't see a way to deal with this unless you are willing to ship your app with thousands of files, most of which will never be used by the device the app is installed on.
I also think it's high time that apple put out an iDevice with the ability to expand memory capacity in the field. I know, I know, they want to control it all. They are big enough to have custom Flash chips made with all the encryption they want so people can't move the flash chips from phone to phone and steal software.
Happy coding.
Why would they run into these realities now? Apple's been around just as long as Microsoft, and they have been pissing off their customer base with poor backwards compatibility the whole time. The scale is much larger now, I'll give you that, but the relentless forward-thinking is what has allowed Apple to get into the dominant position they are in now. Microsoft by contrast, got bogged down with the enterprise where large IT departments put them between a rock and a hard place by refusing to upgrade if they didn't support whatever byzantine legacy application infrastructure they were running. Even to this day where Microsoft realizes they need innovation, they are still forced to hedge their bets with this hybrid Windows 8 Metro crap.
Mark my words, this problem of app size is a tempest in a teapot. Given current trends, most iPads will be 3rd gen within a year, and in a couple years, the first iPad will fall off the iOS upgrade cycle. For every customer that Apple loses because they demand backward compatibility, they'll gain ten with the latest new shiny.
The reason Apple will not do anything this problem is not out of planned obsolescence or greed. All else being equal they would be happy for you to have unlimited space on your iPad. But to solve it is non-trivial and does not push their product forward. To the contrary, it adds complexity that will inevitably slow their product development, which runs directly contrary to their modus operandi.
What I am saying is that these issues will become more and more pressing now that they are operating at a much larger scale than in prior years. In the past they could piss-off their cult followers and move on. I am not sure this is the case today. I think it might be reasonable to assume that normal folks --the bulk of the people buying iDevices--, not tech guys or cult members, will not take kindly to their 16 GB iPhone becoming obsolete simply because Apple released an iPad with more resolution. They would feel that this is absurd, and rightly so. There's nothing whatsoever wrong with older devices. Techies are different. I buy crap I don't really need just 'cause it's cool and sometimes because I want to actively support the company doing the cool stuff. Normal folk are far more practical than that.
If a normal person has a lesser-storage iPhone and, overnight, half their apps go away because they wont fit, they will not run out and buy a new phone. They'll be pissed. This will be particularly true if some mainstream media outlet grabs ahold of the reasons that led to this and outs the story to the general public.
There's another angle to this as well: Trash. Now Apple is in a position to generate, quite literally, mountains of trash based on decisions to not support older devices. I am not an environmental extremist by any measure, but I certainly don't like the idea of millions of perfectly good devices ending-up in the trash bin due to a bad tech decision.
No, it's trivial. Device-dependent DRM-signed packages. Either the common part is duplicated in each device's version, or app store downloads can consist of two packages.
That is not sufficient. You also would have to detect the case where a backup is restored to a device with a different resolution (could even be a lower resolution)
"you will have to foot the bill for your free graphics-intensive app getting three million downloads."
Not necessarily. The App Store could have provisions for doing these kinds of downloads. Because of the backup issue, I even think it would have to have support there, given that Apple would want to prevent the cases "sorry, cheapGames.com's server is awfully slow" or "sorry, junkSite.com's server has disappeared; we cannot get you the graphics for that game you bought anymore" (site names made up; if they exist in real life, the name match is coincidental)
Vector graphics help in some cases.
This isn't necessarily a bad thing. There's a huge tradeoff between size of the product and development speed. We've switched to a pattern of bundling lots and lots of large libraries with everything we ship, and tons of static resources, and tons of translations, and on and on. And it makes everything huge.
But it means you "full-stack engineers" can knock out full-featured, money-making web apps in hours/days/weeks instead of weeks/months/years. Most of the "look at the monetized product I wrote this weekend!" posts that show up on HN could be a fraction of their size, but would have taken much longer to develop.
The link is specifically complaining about resource files, which are probably easier to manage than libraries, but it's still probably not worth the effort for Apple. By making it as simple as possible, developers don't waste any time worrying about how they are going to structure resource usage for their app. Just throw everything in there and get on with it.
At work, I develop for platforms with <1MB of RAM, and we can do plenty with them, but development time is ridiculous. You can knock out a full online dating site in 4 hours, eh? It takes that long to print "Hello World" on a serial port of a new microprocessor... maybe longer.
My side projects are in horribly wasteful monstrosities like Python and Objective-C/Foundation because, in my spare time, I like projects to actually get finished... to hell with binary size.
I would go even further: They wouldn't even change it if it were free and easy to do. If they filtered out Retina graphics for non-Retina devices, the old iPad could hold more apps than the new one! If you upgraded to the new iPad, you would have to trim down your collection. It just doesn't make any business sense for them.
I guess everyone with a jailbreak will rejoice, this sounds like something a Perl one-liner could fix over ssh. And it could remove the builtin apps while they're at it. I am honestly jealous.
I don't mean to assert that Apple shouldn't attempt to separate out Retina assets, but I think it's worth considering the opportunity cost. There are a lot of other things Apple's engineers could be working on. Should this be their top priority?
I've looked briefly into stripping an app: the ipa format is just zip. There is a plist with hashes of the assets. Don't know if that plist itself is hashed, but I'm thinking probably.
Some of the biggest apps on my iPhone are iBooks (57.9MB) and Nook (51.3MB). I doubt their pretty little icons take up the lion's share of that size. I mean, Kindle does it in a measly 23.5MB (sarcasm, still big). I wonder, for instance, how much language support weighs in at. There was a mac program I used some years back called monolingual that would remove other languages, you know because I'll never need my laptop to work in Farsi, and thin the binary architectures. Saved many a GB when they were much more precious.
I'm assuming that most of these apps are made with some fancy service that generates an app for every platform under the sun from one app creator program, and as such, each native app is way bloated.
But, the consumers don't notice or care, so app creators are just going to be as inefficient in the interest of jamming new features into an app over optimizing it.
(edit: spelling, added parenthetical to Kindle app)
iBooks used to be under 20MB. Over time it's expanded to its current size of 50.8MB when it's uncompressed (the extra space it's taking up on your device is probably the books themselves).
The vast majority of that is images. Apple have at least been somewhat smart about it, choosing in certain cases to use lossy JPGs rather than PNGs where the user won't notice (the startup image, for example).
The app also ships with some custom fonts, which take up a few of those MB. The languages, which you suspect take up a lot of the space, actually don't represent that much: each language is about 45KB.
The executable itself is 25MB: so basically, 50% of the app size consists of assets. The percentage is probably higher on most other apps, because the iBooks executable is rather large (unusually so).
From one app developer to another, if you're adding retina assets for the new iPad like me, remember to use ImageAlpha first!
P.S. If you're unsure of its advantages, check out this case study of ImageAlpha + ImageOptim used on Tweetbot: http://imageoptim.com/tweetbot.html
As a developer, I've been more and more often leaning towards using tools like paintCode or just writing code for custom widgets instead of trying to make the 4 different images for iphone/ipad/iphone@2x/ipad@2x. This is a win/win because it looks better (except at very low res) and keeps the size of the app down.
Having the app store serve different apps for different target devices opens a can of worms. For example, if you backed up on one device and restored on another, you could get the wrong one. I think Apple made the right choice in preferring to make sure your apps work everywhere rather than save storage at the cost of complexity. And if that means they're nudging the consumer to upgrade to the lastest and greatest, I can't blame them for going that route.
But as I've run down on space, I've started to notice how odd various app's sizes are. There seems to be very little correlation between what I expect and the actual value. Here are some examples from my applications:
First, some apps that have sane sizes
Infinity Blade - 1 GB
Rage - 830 MB
Ghost Trick - 531 MB
Groove Coster - 154 MB
Steambirds Survival - 60 MB
Doodle Jump - 20 MB
Worms 2 Armageddon - 688 MB
Worms - 209 MB
Catan - 171 MB
Boggle - 104 MB (really? REALLY?)
XFinity (Comcast App) - 41 MB (has downloaded another 50 on my device)Most apps' size can be halved this way: http://imageoptim.com/tweetbot.html
I ask because I was thinking of getting an iPad3 in the near future and was tempted to get the 16 GB model because I have many TBs of storage elsewhere on my PC/NAS/Servers so I'm not going to use it to archive anything long term. Plus an extra £70 for 16GB seems steep when I could get a 64GB SSD for the same.
Consider download complexity. How many "versions" of your app would you need to have to give each user the optimal installation size? How often do you want to deal with people downloading the wrong one and having a bad experience? What about 3rd party distribution...any number of other sites may choose to offer download links and you have basically no control if they screw something up.
Another is problematic upgrades. Suppose you optimize a download for a user's current machine. Then they buy a new machine. They won't really try to understand why your app is now looking strange or misbehaving; they'll just think your app sucks.
On the Mac side I've chosen to go the oversized route. There's no doubt that my app bundle is 2-3 times as big as it ought to be (and unfortunately two thirds of the binary bloat goes towards supporting systems that few users are likely to care about at this point). But then downloads are simple: just "click here for the new version" and whatever your system you see something that works.
Over the next few years, we'll probably go back to having just two screen sizes (Retina iPhone + iPad). This is just a side effect of the transition to retina, and it will go away soon.
Though from Apple's perspective it would make more financial sense to have you go buy a 64GB iPad. :)
You're probably going to get very comfortable with this settings at some point, if you don't have a 64GB iPad.
To an embedded developer, the hype about 'the desktop is dead' and 'all apps must go portable' was ludicrous. Speed, space, resolution, connectivity all suffer in the mobile environment. No amout of cool design will fix this, wishful thinking aside.
Poor developers! You have to work hard to fit into the space available. And maybe abandon wasteful tools that squander memory and cpu, in favor of tools that make you face the hard choices. Guess what? Its called software engineering, and you're supposed to have the skills to do it. Stop complaining!
Instead of using a CDN and eating the bandwidth cost of retina display-ready assets, it sounds like apps companies are publishing the cost onto users for increasingly resolute image assets.
That or people should buy Apple's larger models if they expect to do serious computing on the iPad.
Instead, just about every response (save one on the actual linked page), addresses the issue at hand. It seems to suggests that there is an appropriate level of sensationalism to get your message heard. Anyone have some sort of algorithm that rates a post for how attention-grabbing vs. off-putting its words are?
Would Apple allow an app maker to release their app for only newer iPads and iPhones, and not for the older devices? (I have no idea...)
What I'm asking for isn't crazy: You can kind of do this today by shipping a separate iPad and iPhone app (you can't separate non-HD and HD). The main thing I'm asking for is for the App Store to recognize that they are the same thing without forcing me physically combine them.
As a matter of price, for the reasons above I am OK to pay more for an universal app than have two different ones I'll have to manage separately.
With Android, there could be any one of dozens of resolutions (iOS has two basic resolutions, with a relatively simple 2x multiplier version of each of those), you could have a keyboard or not, you could be a device with a resistive screen with no multi-touch (or even no touchscreen at all), your OS version may dictate whether you can install on the SD card or not - or have one of many other significant differences - and the device may not be able to upgrade to a more up to date OS version, etc etc.
When we launched our app recently, we tested it on 3 different iOS versions (a 3GS, a 4S and an iPad 2), and about 20 different Android ones. I can't remember a single major issues that appeared on only one model of iOS device - and I can't think of any that have appeared in the wild either. On the other hand, we had dozens of "only occurs on Android device X with OS version Y".
We're currently seeing around one new iPhone and one new iPad coming out of Apple each year, though. Don't forget iPod Touch, by the way. Sometimes they come with new features not available to older devices. They get released with three different storage capacities, and only some of your users will have 3G/4G on their iPads. The point is that in a year or two this is going to be a worse situation even for iOS devices. It's reasonable to assume that Android fragmentation will also get worse.
Where does that leave us? Well, I think Android is taking an interesting approach. A couple of the things in the Android SDK that help deal with fragmentation: - Standard UI components (eg. Action Bar for actions/navigation, Fragments for generic slices of UI, Layouts for providing guidance on how Fragments are to be arranged on different devices and screen orientations) - They provide a support library that can be bundled with apps using features from newer OS versions so that the new features also work on devices running older versions.
According to the ZipLine CEO here:
http://thenextweb.com/google/2012/04/02/zipline-ceo-stop-whi...
most of the issues they saw along those lines were general problems that happened to be manifesting in a particular situation. I'm curious about whether or not that experience is typical or, for instance, if ZipLine saw that because they were working on a framework first and a specific game second.
The vast majority of normals will never even realize this is the case. Plus, even if they do, people are well conditioned to the obsolescence of technology. I mean more people buy a new PC before they re-install Windows when things get unbearably slow, and if we had a graph of how often people upgrade their phones, I don't think we'd see much to support your point.
Now if the media decides to latch onto this (which I wouldn't doubt given the fervor for a juice Apple scandal), then it might piss off a few people, but do you really think that people will be convinced to trade in their iPads for pitchforks over this? What percentage of people are really not going to be able to update their apps anyway? If this really became a media problem then Apple might move to address it, but I stand by my written and attributed assertion that this is a non-issue for Apple and will not affect their bottom line or move them to any action other than to perhaps go 32/64/128 for the next iPad.
You are committing the common developer error of underestimating complexity. But when you get in there, I assure you, having one package per app is dramatically simpler.
Prepare two images - 512x512 and 32x32. Scale first one down to 32x32, subtract it from the second one and this will yield a arithmetic, per-pixel difference. Compress it. Now store 512x512 image and the compressed diff with an app. With these two, you will then be able to recreate original 32x32 (by scaling down 512x512 and applying the diff). And I am making an (educated) guess here that the compressed diff will take less space than the actual 32x32 image.
I see people suggesting going with SVG or resizing a high resolution image, but there was a good post not long ago about the problem with SVG and blindly resizing. You really need your images tweaked on a per-size basis.
I can imagine you can get away with automatically resizing iPad HD -> iPad, but going down to the relatively tiny sizes for the smaller screens would be a bad idea IMO.
True, and false. All you would need is a vector graphics format with a per-element LOD specification. Here's an example of one format and editor which does that:
http://www.haiku-os.org/docs/userguide/en/applications/icon-...
Can Android say the same thing?
I agree that it's a bit of a pain having to have both the low res and high res graphics bundled in a single app version, but this issue isn't one of legacy - it's the same on a 2 year old iPad 1 as it is on a brand new iPad 3. If you've only got 16GB on it, you're possibly going to run into space issues with these new larger apps.
Things like storage capacity and network availability are challenges, it's true. But they are things that can affect any device (you can fill up a 64GB device, and you can be in an area with no network), so they aren't really fragmentation issues unless your app requires huge storage or depends on a specific network type being available. Android by its very nature is always going to have a far wider variation than iOS. This is not necessarily a bad thing (there's more choice and more cutting edge options in Android), but it does create a fragmentation challenge of a totally different scale to iOS.
Name a single one an end-user would encounter. They wouldn't.
However even if it is a bug, bugs do happen and if it's a bug that only shows up on one out of 20 OS/handset/network combinations (if you're lucky enough to have that one specific combination in your test labs) is going to be one that's going to be a lot harder to spot than one that either occurs on all devices or not at all.
After all, when it is a general bug that just happens to be masked most of the time, it is a buried bomb just waiting to explode when the surrounding context changes. In my opinion, detecting and defusing more of these issues sooner rather than later is a good thing.
Personally, I'm reminded me of the issues I run into when writing cross-platform code (or code that needs to work with multiple versions of tools and libraries). The upfront effort to multiple environments (or the effort required to add the new environment) is usually significant, but I'd say the improvement in code/software quality is an often unrecognized benefit (over and above the reason you're supporting the new environment in the fist place).
And it's bloody inconvenient for the user who has to buy and keep half a dozen identical applications.
Right now the situation is: toss whatever you want into the bundle and it's there everywhere. For what you're proposing to have any benefit it would have to be: add files, annotating them with whether they are needed for iphone/ipad, keeping those annotations updated as you work.
When making a universal app you are already forced to do this. Take a look at the Twitter app: it is completely different on iPad vs iPhone. These labels are how it knows what interface to load on each device -- yet the code and resources to load either exist regardless of what device you are on.
Games are by far the worst of course, but I'm sure part of that is down to the higher-res textures for iPad2/3 which are useless on the first iPad's much weaker GPU. (the difference is likely even worse between iPhone4S and iPhone 3GS or older) Adding a higher-res mip level for each texture in a game will increase the size of all textures by about 155%[1]. So in principle a 16GB/32GB non-retina iPad could store at least as many apps as a 32GB/64GB retina one, respectively. (not counting savings from losing all the iPhone/iPod touch assets, which will be big for non-OpenGL apps)
The iOS app update mechanism is also pretty broken when you're low on disk space: to update large apps, you have to delete the old version (deleting any of your data in said app) and reinstall with the new one, if both of them don't fit side-by-side on your device.
And as an app developer I'm already so used to jumping through ridiculous hoops to get apps onto the app store, so tagging files for different device types seems straightforward enough (and would help a LOT for reducing the size of one of the apps in the store that I developed - in fact, pushing it below the 3G threshold vs the current wifi-only size).
And it works fine except for the fact that it chooses the resource at runtime. When you ask for resource "x" it looks for all the resources that start with "x" but may end with @2x or ~iPhone or whatever. It then decides which to return based on what device you have. There is no reason it cant make this determination at compile-time and just not include the resources it knows it won't be using in four separate builds. There is literally no difference with this system except when it chooses to ignore resources.