I built an AI calorie tracker, so take this with whatever amount of salt you think is appropriate. But I've spent hours upon hours of my life figuring out the problem, and I think most of what gets written about it is either marketing or dismissal, with not much in between. This is an honest take.
The two jobs
When you send a photo or text to the AI, it runs through two jobs: identifying the food and estimating the portion. The former is easy now. After researching for months, I could only find older models capable of identifying a subset of foods. Some models could do a better job, but the subset had to be predefined. Modern LLMs, on the other hand, can identify more food types than most people probably could, though they sometimes struggle with certain lighting or angles. In my experience, these edge cases are few. Often, they can even infer the cooking method or ingredients you probably had in the dish, and they'll be right a lot of the time.
The second part, portion estimation, is trickier. Because your image is a flat visual of the food containing no volumetric information whatsoever, the AI has nothing to reference. It can guess portions based on the typical sizes of bowls, plates, forks, or something else in the image, and it does a decent job. But this is where nearly all of the error comes from in any AI food scan.
There are some methods available that can accurately estimate the portion of food from an image, but they're computationally expensive, slow, not consumer-friendly, or all three. That's why no one uses them seriously in commercial products. Yet.
When researching, I honestly expected AI portion estimation to be nearly solved, considering the practical applications, but it hasn't been. Still, time and time again I'm surprised by how well LLMs infer portions from just an image. In my quick assessments, it's within the same margin of error as my own estimates, and I've been tracking food for years and have the food right in front of me. It's impressive, but not perfected yet. It's rarely wildly off, but that's why I check its work and swap out ingredients it didn't get quite right.
Anyone telling you portion estimation from a single photo is solved is selling you something.
Where it's accurate and where it isn't
In terms of identification and portion size, AI will almost always have good or, at worst, decent accuracy for a plated dish with clearly visible ingredients. It can often reliably detect the cooking method, use its vast knowledge of cooking practices to its advantage, and use everything around the plate as a reference. It's typically clean. The only caveat is if you take the photo from a deliberately difficult angle, like top-down, where there's little depth information to help. It can still identify the food, but its portions will be off unless it's a standard-size item.
If you have food in a bowl, sauces that aren't visible or traditionally used, or something amorphous like pudding, cereal, or oatmeal—it will struggle. Sometimes it can guess well, but often you'll need to revise based on how much of each ingredient you actually used. Sometimes it even misses ingredients if they're at the bottom of the bowl. It's not the end of the world, since you can always revise it. At worst, it can infer from typical preparation and bowl size, which isn't bad. But the point is: it's good, but there are limits to its accuracy.
I almost always weigh my cereals or give precise volumetric portions of my oatmeal (e.g., 1 cup dry rolled oats) to the AI when I'm telling it to log something for me.
How SavorAI handles it
The app we made plays to the strengths of modern AI I outlined above rather than its weaknesses.
In the app, you can send either a photo or text/voice input for analysis. As usual, it identifies the foods to the best of its ability. Then it assesses the cooking method, preparation, and whether there are any additives it needs to add to the ingredients list (like salt for a home-cooked meal). It also uses the user's locale to check for relevant food types. It then searches the database based on these specifications. Most items come from the USDA or other compiled databases we've put into the app, meaning they're lab-verified entries instead of AI estimations. Database matching fixes the nutrition data, but the portion is still an estimate—which holds up well for many foods but not others.
For those others, the context feature is the fix. Users can specify precise volume or weight, cooking method, or the portion eaten. The AI can take this and use the USDA modifiers to conclude that "1 cup of 2% milk" with someone's breakfast is 244g, so it puts precisely that number in the food log, making it as accurate as any manual logging would be.
Still, even lab-verified data will be very slightly off. So are food labels. This isn't an AI limitation; it exists in all food tracking, unless you're eating the very same chicken their labs used for these measurements.
Not likely.
That's why this pipeline, used as intended, is faster and more efficient, with the same or sometimes better accuracy.
Manual tracking isn't a clean comparison
Logging by hand is the same two jobs, but the person logging often has worse information available. Unless you weigh your food with a scale, people's estimates (especially without experience) tend to be on the low end, meaning they're eating more than they think. Because of this, the model is often more accurate at portion estimation than the average logger.
Even when I test my portion estimates against the AI's, it's sometimes closer than I am. Other times I'm closer. Again, I have years of serious, weighed logging behind me. I eyeball well.
So the real comparison is against what you'd actually do otherwise. Weighing your food is as close as it gets, and you can still tell the AI the weights by voice or text, and it'll search the database faster than most people can.
Getting better numbers out of it
This is the baseline advice I'd give anyone wanting to use an AI tracker:
- Barcode or label scan when one exists. You'll get the numbers directly from the manufacturer.
- Use the context feature when you know the portion. Even "1 small bowl" or "3 spoonfuls" helps.
- Shoot at an angle with a size reference in frame, like a utensil or a standard object. You don't have to go out of your way for the latter, but keep it in mind.
- Revise and correct the estimate when you know it's wrong. If it added an ingredient, omitted one, or got the portions off, you can tell it so, or modify the ingredients manually yourself.
- Weigh the foods you eat daily once, then repeat the entry. If your app supports a repeat feature like ours does, this is easy. Otherwise, save it as a custom entry and log it daily.
- Log the oil and butter. They're very calorically dense.
Read this article for more tips and commentary on this subject.
Conclusion
AI calorie tracking isn't yet advanced enough that you can have an AI estimate everything and hit your goals precisely. You'd get close, but it's better to have more precise numbers when you're serious about weight loss, weight gain, or body composition goals. Especially the latter. What matters more is consistency. If you're consistently weighing most of your food and estimating some, you're going to do fine in most cases. Stick to scanning labels and barcodes when you can, add context to your scans, correct what you know, and log the fat in the dish.
Then you'll do great!
