By Robert McMillan

IDG News Service – More than three years after the iPhone was first hacked, computer security experts think they’ve found a whole new way to break into mobile phones — one that could become a big headache for Apple, or for smartphone makers using Google’s Android software.

In a presentation set for next week’s Black Hat conference in Washington D.C., University of Luxembourg research associate Ralf-Philipp Weinmann says he plans to demonstrate his new technique on an iPhone and an Android device, showing how they could be converted into clandestine spying systems. “I will demo how to use the auto-answer feature present in most phones to turn the telephone into a remote listening device,” he said in an e-mail interview.

Weinmann says he can do this by breaking the phone’s “baseband” processor, used to send and receive radio signals as the device communicates on its cellular network. He has found bugs in the way the firmware used in chips sold by Qualcomm and Infineon Technologies processes radio signals on the GSM (Global System for Mobile Communications) networks used by the majority of the world’s wireless carriers.

This is a new area of research. Until recently, mobile phone attacks had focused on another part of the phone: the programs and operating systems that runs on the device’s CPU. By tricking someone into visiting a malicious Web site, for example, hackers could take advantage of a Web browser bug on the phone and start messing around with the computer’s memory.

With baseband hacking, security researchers are looking at a brand new way to get into this memory.

“[It’s] like tipping over a rock that no one ever thought would be tipped over,” said the Grugq — a pseudonymous, but well-respected, wireless phone hacker, and one of a handful of people who have done research in this area. “There are a lot of bugs hidden there,” he said, “It is just a matter of actively looking for them.”

But hacking a smartphone with a baseband attack is very tricky, to say the least. The mobile phone’s radio communicates with a cell phone tower. So in Weinmann’s attack, he has to first set up a fake cell phone tower and then convince his target phone to connect to it. Only then can he deliver his malicious code. And even then, the malicious code he writes must run on the firmware that’s used by obscure radio processors — something that most hackers know nothing about.

“This is an extremely technical attack,” said Don Bailey, a security consultant with Isec Partners. He says that while the work on baseband hacking is very exciting — and ultimately a big deal for the mobile phone industry — he doesn’t expect any attacks that target the general public to emerge anytime soon.

But the research into this area is just starting to take off, fuelled by new open-source software called OpenBTS that allows virtually anyone to set up their own cellular network radio tower with about US$2,000 worth of computer hardware.

Five years ago device makers didn’t have to worry about this type of hacking, because it used to cost tens of thousands of dollars to set up a cellular tower. But OpenBTS has changed all that. “Now it’s a completely different game,” Bailey said.

It’s a risky game too. In the U.S., federal wiretapping laws make it illegal to intercept phone calls over the licensed frequencies used by mobile phones. In August, it took intense last-minute negotiations between lawyers from the Electronic Frontier Foundation and the U.S. Federal Communications Commission before security researcher Chris Paget could demonstrate a very simple tower spoofing technique at the Defcon hacking conference in Las Vegas.

Two months from now another hacker conference, Vancouver’s CanSecWest, will invite hackers to break into mobile phones using a low power transmitter. If their baseband attacks work, they can win cash prices. Conference organizer Dragos Ruiu said that Canada’s broadcast laws are “more lenient’ for researchers who want to set up low-power towers for research purposes.

Still, it remains a touchy subject. “Last year we were worried about falling afoul of regulations,” he said.”Now we’ve figured out a nice safe way to do that so that we don’t mess up anybody else’s cell phones at the conference.”

Ruiu expects some interesting results from the contest, called Pwn2Own. “It sounds like the radio parts of the phones are very shaky indeed and pretty vulnerable,” he said.

Robert McMillan covers computer security and general technology breaking news for The IDG News Service. Follow Robert on Twitter at @bobmcmillan. Robert’s e-mail address is robert_mcmillan@idg.com