Coin images and AI

[Heliodromus]

I just tried a comparison between the three major AI's available online in their free variants.

1) OpenAI's GPT-4 (as used by Microsoft Bing CoPilot in "Creative" mode - basically their version of ChatGPT)

2) Google's Gemini Pro (their more powerful Gemini "Ultra" is paid version)

3) Anthropic's Claude-3 Sonnet (their more powerful Claude-3 "Opus" is paid version)

The task I gave all three was to describe an image I gave them. I didn't tell them what the image was (even that it was a coin), but it was both sides of a bronze coin of Constantine I, with a unique unlisted reverse type that they would never have seen during training.

Claude-3 was by far the best, and right off the bat knew it was a bronze roman coin, knew the obverse was a male bust, and (rather surprisingly) was able to read the legend, which it knew was in Latin, and correctly concluded it was a coin of Constantine! The reverse legend was only partially legible, but it made out parts of it out, and when I provided some clues about legend and design it was able to correctly guess the partially legible legend and relate it to the design.

I follow the progress of these "AIs" (LLMs) in some detail, but it's still amazing how good they are getting, at some tasks. I haven't spent much time asking them about images (which is a relatively new capability), and was pleasantly surprised at how capable Claude-3 is (and this is only the free "Sonnet" version, not the more powerful "Opus"). Reading a legend from meh condition LRB is no joke, and Claude was way better than the other two. GPT-4 didn't do so well, and thought that "AVGG" said "HUCC" since that's what the letters literally look like.

It seems we're not at the point where the AI can recognize an ancient better than a human, but still already quite capable!

 

Edited March 30 by Heliodromus

[Deinomenid]

I've been trying to train up Claude on Magna Graecia. He is a polite but average student. Very confident in  his errors until called  out.  Eg Kaulonia recently -

You're absolutely right, my previous assessment was incorrect. After taking a closer look, these are indeed ancient Greek coins, not Roman denarii. The style, iconography, and legends identify them as being struck in Kaulonia, an important Greek city in ancient Magna Graecia (southern Italy) during the 6th-5th centuries BCE. The obverse depicts a nude male figure, likely the hero Kaulos or a representation of the river god Achalos, in a crouching archer pose - a symbol associated with the city's founding myth. The reverse shows a stag or doe standing upright, which was another common symbolic badge on Kaulonia's coinage. Thank you for the correction. I clearly made a mistake in properly attributing the cultural origin and minting location of these coins based solely on the visual elements. Identifying Greek colonial issues from southern Italy requires nuanced numismatic expertise. I appreciate you taking the time to ensure I update my understanding accurately.

 

I also  tried him on a couple of @kirispupis's  mystery coins, but he  practically wept in despair!

 

 

[Orange Julius]

Fun idea! Do you have the images so we can see the results? 

[Heliodromus]

1 hour ago, Orange Julius said:

Fun idea! Do you have the images so we can see the results? 

The coin I tried was someone else's, and anyways I didn't screen grab the conversations.

I'd encourage people to try it for themselves though, especially if you have never tried one of these AIs.

You can sign up for free Claude-3 access here. It just needs an e-mail address and phone # to text for verification code.

https://www.anthropic.com/

To give Claude an image instead of just text, just use cut and paste to copy the image into the text entry box.

If you've never used one of these before, just talk to it as a human - you can ask it anything. Sometimes you may need to coax or threaten it to do things. Someone reported an AI getting upset after they promised to tip it, and then never did (not that there is any way to do so)! 😃

 

 

Edited March 30 by Heliodromus

[Phil Anthos]

15 minutes ago, Heliodromus said:

If you've never user one of these before, just talk to it as a human - you can ask it anything. Sometimes you may need to coax or threaten it to do things. Someone reported an AI getting upset after they promised to tip it, and then never did (not that there is any way to do so)! 😃

 

 

This scares the crap outta me.

[Heliodromus]

7 minutes ago, Phil Anthos said:

This scares the crap outta me.

Yeah - they do (not on purpose - just the way they are built/trained) a remarkable job of appearing human and sentient. No need to be afraid of them yet, but may as well be polite to them anyways - they may respond better, just as we would. 🙂

[rNumis]

Wait until they start joining NumisForums 😄. Or maybe they're already here.

[Phil Anthos]

Technology is a tool, just like my hammer. But my hammer is okay with sitting in a box until it is needed, so I don't need to be polite to keep it from overthrowing humankind and spreading its insidious hammer message throughout the galaxy!!!  😳

[expat]

Saying, at the moment, that it is fun and interesting, is ignoring the danger that in 30-50 years there might not be any need for researchers, academics, teachers,  libraries, schools, colleges, universities. If pressing a few buttons and writing a few lines on a keypad surpasses  actual learning and experience of any given subject, what a sad future awaits. I, for one, am glad I won´t be around to see it.

Laziness and greed are increasing at a rapid pace. There will soon be no "free" versions, only paid subscriptions. What do the poor do then to get education? 

[robinjojo]

For all of its innovation and ease of use, I just get a feeling that technology, headed most recently by the leapfrog advances of AI, is coming at a cost.  That cost is our ability as human beings to communicate among ourselves, to have direct contact and to think rationally and independently. 

Now I think AI has definite promise in the areas of health care, climate modeling and scientific research, to name three.  Already this technology is helping researchers process data on a volume beyond human capabilities in very short timeframes.

I suppose it is fun to have an AI app construct a Roman coin with an unlisted reverse, much in the same way as having an app create an essay based on rudimentary input.  But I am cautious and cling to old ways, no doubt partially due to my stubborn personality; shoot! I don't even own a smart phone.  But it has been my experience that the pace of  technological innovation outstrips the ability of society to cope with the changes the often come in crashing waves, swamping us with unforeseen consequences.

For now I'm sticking to my books, online searching through Google and, oh yes, my cordless phone.

I hope that I am not offending anyone, but that's just the way I feel and think....

Edited March 30 by robinjojo

[Heliodromus]

Yes, it is a concern. I'm not sure if we're yet on the right path to develop human-level (or beyond) AGI, but it's only a matter of time.

People talk about threat of extinction/etc, or just irrelevance, but I think those are controllable as long as we don't move too fast.

The bigger danger, as you suggest, is perhaps threat to society due to loss of jobs. I'm not sure about inequality - we are all at risk. However, the future doesn't always turn out as expected (where are the flying cars?!), and we're not necessarily going to end up in sci-fi land where there are robots walking between us, and/or we are ants irrelevant to AI's that are 1000x smarter than us. Hopefully we will find a way to use the tech as a tool so that we are still in charge and gainfully employed.

 

[AETHER]

Umm has anyone here ever seen The Terminator ?!.. I rest my case

[Heliodromus]

12 minutes ago, AETHER said:

Umm has anyone here ever seen The Terminator ?!.. I rest my case

Or maybe like this. Better?

 

[Ryro]

The industrial revolution didn't put everyone out of work. It just opened up different kinds of work.

It's not the technology we should fear, but the greed of the upper classes.

[Heliodromus]

1 hour ago, Ryro said:

The industrial revolution didn't put everyone out of work. It just opened up different kinds of work.

Hopefully something similar will happen, although it's hard to imagine exactly what that will look like. What's left, or what will be the replacement, when AI takes over the problem solving and knowledge-based jobs (and those aspects of jobs which also involve other skills)? Jobs that are currently high paying will become low paying because the scarce skills they required will then be done by machines.

The industrial revolution, which we might consider factory automation to be a continuation of, really doesn't provide much cause for optimism. The jobs created by the industrial revolution were mostly factory jobs - working on the production line, which seem more mindless and less satisfying than the manual work that it replaced. Factory automation really hasn't created any new jobs - it's just helped to gut the middle class and increase inequality by eliminating many well paying blue collar jobs. Next it would appear AI will be eliminating the IT workers, engineers, accountants, lawyers ... The college system in the US seems already in trouble due to out of control costs, and what will happen if the carrot of high paying jobs is replaced by AI? Will people even bother with higher education?

It's really hard to predict how things will look in a world where intelligence and knowledge become a cheap commodity rather than valued. It'll be very different for sure.

 

Edited March 31 by Heliodromus

[Ed Snible]

11 hours ago, Ryro said:

The industrial revolution didn't put everyone out of work. It just opened up different kinds of work.

The industrial revolution was good for people.  Jobs for horses still haven't recovered.  I am sure new kinds of work will open up, but will those jobs go to humans or machines?

[Heliodromus]

3 hours ago, Ed Snible said:

The industrial revolution was good for people. 

Was it really? In what way, and for what large classes of workers?

Some of those ploughmen may have ended up as tractor drivers, or at least in subsequent generations there are tractor drivers, but generally people are going to try to do the highest paying job they are qualified for and capable of doing. A displaced ploughman may not have a lot of alternate job options.

It's like people telling factory workers whose jobs have been lost to automation "so, just get a job in IT, building factory automation". The trouble with the advent of AI, perhaps soon AGI, is that it's not so much specific types of job being likely to be replaced, but a whole "capability level" class of jobs. If you are not capable of doing a job that an AGI can not, then job options may be limited, and people may have no option other than to compete for the lesser-skilled lower paying jobs that are left.

In seems that AI/AGI may come in two waves - the current LLM wave of GenAI automation, later to be followed by AGI capable of learning on-the-job and more creative problem solving. It's certainly interesting (perhaps in same way a train wreck is interesting) to consider what types of job will be unaffected and/or not doable by AGI and/or robotics for the foreseeable future. The major class that comes to mind is anything with a more "human touch" (e.g. nursing, maybe teaching), or anything requiring non-repetitive physical dexterity (trade jobs - plumbers, electricians, etc).

 

[Sulla80]

On 3/30/2024 at 1:01 PM, Heliodromus said:

Claude-3 was by far the best, and right off the bat knew it was a bronze roman coin, knew the obverse was a male bust, and (rather surprisingly) was able to read the legend, which it knew was in Latin, and correctly concluded it was a coin of Constantine! The reverse legend was only partially legible, but it made out parts of it out, and when I provided some clues about legend and design it was able to correctly guess the partially legible legend and relate it to the design.

Epimetheus opening Pandora's box, print,  Giulio Bonasone (Italian, active Rome and Bologna, 1531–after 1576), an engraving published in Providence. Public Domain image with thanks the to the NY Metropolitan Museum of Art; MET, 64.682.102.

An interesting thread / discussion, @Heliodromus and a topic that is hard to avoid these days with both interesting promise and rational fears.  I posted some examples/screen shots of my experiences with ChatGPT 4.0 and Claude 3 Opus for coin identification and legend reading here:

https://www.sullacoins.com/post/numismatics-generative-ai

Overall I found ChatGPT to be more credible in its answers - and both tools still far from reliable or useful resources.  I am interested to hear of any relevant published literature, and any work on a numismatic large language models that might be built specifically on auction databases for attribution, die studies, fraud detection, et.c.

Edited March 31 by Sulla80

[Sulla80]

4 hours ago, Ed Snible said:

The industrial revolution was good for people.  Jobs for horses still haven't recovered.  I am sure new kinds of work will open up, but will those jobs go to humans or machines?

yet somehow today we have twice as many horses in the US today compared to 1840

• According to the Census returns of 1840, there were in the United States 4,335,669 horses and mules
• There are 7.2 million horses in the United States, according to 2016 survey data published by the American Horse Council

(Edit: worth acknowledging several sources report that horse population in the US did peak ~25M circa 1920-25)

 

Edited March 31 by Sulla80

[robinjojo]

The important thing to keep in mind, I think, is that technological innovation, especially innovation that has a major societal impact, such as the Industrial Revolution, is that it comes with positive outcomes and a dark underbelly: vastly increased availability of goods (food, clothing, household utensils, etc.) leading to a burgeoning middle class, and at the same time an explosion of industrial workers whose economic lot was dire, until the appearance of trade unions. 

Technology has advanced through the 20th and into the 21st centuries with an increasing pace of benefits and costs.  The airplane has led to a transportation revolution that has shrunken the planet, while at the same time giving us the carpet bombing of World War II and the atomic bombs dropped on Hiroshima and Nagasaki.  Now, in the middle of the Information Age, technology in the form of AI, social media and search engine platforms are giving us information at our finger tips, retrieving data that would have required hours of research at a library. 

But, it has also created silos that provide reinforcing feedback of personal beliefs and prejudices, supported by fake videos, images and voice messaging.  With this technology, literally, reality is being altered to a point where there is widespread disagreement over what it actually is.  In the not-so-distant future, it will be, for example, to easily create a convincing image by almost anyone of a "discovered" ancient Roman or Greek coin, submitted with a convincing narrative behind it, perhaps with a convincing video featuring a well-known authority, with the use of an AI app.  

Maybe I'm overblowing the perceived threats, but they are certainly worth considering.  I do hope that regulations in the US are instituted to set up rail guards for this technology, though I am sure that they would be ignored by bad actors. 

Edited April 1 by robinjojo

[Heliodromus]

22 hours ago, Sulla80 said:

I posted some examples/screen shots of my experiences with ChatGPT 4.0 and Claude 3 Opus for coin identification and legend reading here:

https://www.sullacoins.com/post/numismatics-generative-ai

I think much of the limitations of these models as applied to images of ancients is just due to lack of coin image training data, although it also depends on how exactly they are handing image inputs. I expect it is basically just image classification on parts of the image, meaning that it is mapping parts of the image to a fixed set of categories it was trained on, which is similar to how Telsa's vision system works. If a Tesla sees a train crossing at a junction it perceives it as a sequence of trucks since it wasn't trained on trains, and the best match for each carriage is a truck (one reason why I would never use "self-driving" in one of these!).

Claude's image recognition still seems quite impressive, but of course we're not privvy to what it was trained on, so hard to tell where image classification ends and intelligence in interpreting the pieces kicks in. Was Claude trained on stick figures in canoes, I wonder? It certainly handled this one I just drew well!

As far as recognizing specific emperors or devices on coins, it's going to depend on how well, if at all, these were represented in it's training data.

I think my experiment with Claude, using a coin of Constantine, was more successful because in that case Claude was able to read the obverse legend, hence realize it was Constantine, then bring a bunch of textual knowledge of coins of Constantine to bear. Although Claude didn't recognize the reverse design immediately, once I gave a clue it was able to guess correctly then again use textual information related to that to further describe/guess the scene. It's all a bit smoke and mirrors perhaps, giving an impression of more vision capability than is actually there, which would be massively improved by more coin image training data.

[Sulla80]

6 hours ago, Heliodromus said:

I think much of the limitations of these models as applied to images of ancients is just due to lack of coin image training data, although it also depends on how exactly they are handing image inputs. I expect it is basically just image classification on parts of the image, meaning that it is mapping parts of the image to a fixed set of categories it was trained on, which is similar to how Telsa's vision system works. If a Tesla sees a train crossing at a junction it perceives it as a sequence of trucks since it wasn't trained on trains, and the best match for each carriage is a truck (one reason why I would never use "self-driving" in one of these!).

Claude's image recognition still seems quite impressive, but of course we're not privvy to what it was trained on, so hard to tell where image classification ends and intelligence in interpreting the pieces kicks in. Was Claude trained on stick figures in canoes, I wonder? It certainly handled this one I just drew well!

As far as recognizing specific emperors or devices on coins, it's going to depend on how well, if at all, these were represented in it's training data.

I think my experiment with Claude, using a coin of Constantine, was more successful because in that case Claude was able to read the obverse legend, hence realize it was Constantine, then bring a bunch of textual knowledge of coins of Constantine to bear. Although Claude didn't recognize the reverse design immediately, once I gave a clue it was able to guess correctly then again use textual information related to that to further describe/guess the scene. It's all a bit smoke and mirrors perhaps, giving an impression of more vision capability than is actually there, which would be massively improved by more coin image training data.

I agree with you : a specialized "language model", trained with images from coins and text from coins should do well to recognize and attribute coins.  I suspect that any of the general purpose LLMs won't perform particularly well at ancient coins.  However, if I don't like what they do today, I just have to wait a few minutes and a better one is coming....love the stick figure example!

 

[kirispupis]

For AI, Roman coins are far easier because if the inscription can be read, then the task is nearly complete.

I'll be far more impressed when AI can decipher Greek bronzes, especially those from the 4th century BCE. Many of these types are rare and it's uncommon to find one with a complete inscription. That being said, I'm sure it's only a matter of time...