Protecting Corporate Assets with ShodanHQ – An Interview with the ShodanHQ Creator
ShodanHQ describes itself as “the world’s first computer search engine that lets you search the Internet for computers” and allows you to “find devices based on city, country, latitude/longitude, hostname, operating system and IP [address]”. Data discovered using ShodanHQ was recently quoted in Senate testimony promoting the CyberSecurity Act of 2012. ShodanHQ creator John Matherly (@achillean) shared some time with us to offer a little information to readers of the Cyveillance Blog.
Cyveillance: How did the idea for ShodanHQ come to you?
Matherly: I thought scanning the entire Internet would be an interesting problem to solve – I thought it would be fun! I had just written a basic network scanner and as I started using it I realized that sharing and indexing those results might be interesting to others. It started as a hobby during college and have rewritten it over the years until I reached the version it’s at now. Originally, I envisioned Shodan as a service similar to Netcraft but it would cover more services and provide greater access to users. My expectation was that market researchers would enjoy Shodan as a source of empirical data on software usage. The security community picked it up instead, and since then it has developed into a global network of servers that collect data in real-time on a dozen services/ ports from devices around the world.
Cyveillance: Tell us about the scope of the data in ShodanHQ. If an average user comes along, how likely is it that the find what they’re looking for if it exists, and how recent would that data be?
Matherly: Shodan currently includes data on the following services:
- Alternate HTTP
- Oracle Web
- MongoDB Admin
Data is constantly collected and on average 5-9 million new records get added to the database each month. Shodan brute-forces the entire IP space to ensure uniform coverage of the Internet and make sure it doesn’t miss subnets due to any algorithm bias. If a device is connected to the Internet, Shodan should have it indexed.
Cyveillance: Would you describe ShodanHQ as a penetration testing tool?
Matherly: It was designed as an intelligence gathering tool, but it gained traction in the penetration testing community. As such I would consider it a penetration testing tool, though it’s best coupled with other tools that can consume Shodan data via the API (see FOCA).
Cyveillance: Let’s pretend I’m part of an information security team at a large corporation. What are the first three queries you recommend I make using Shodan to help protect my company?
- Look at the Most Popular Searches on Shodan from your dashboard and select a few of them to get a feeling for how Shodan works.
- Run a search using the ‘net’ filter, where your network IP range is provided as the argument (ex: net:126.96.36.199/24).
- If your company provides a product that could be facing the Internet, search for it on Shodan. Depending on the product you can identify misconfigured devices, where they’re located and what version is most popular.
Cyveillance: Much has been written about internet-based vulnerabilities found in civil critical infrastructure environments like water and electrical power. Based on what you have seen in ShodanHQ, how real is the threat? How insecure are these SCADA systems?
Matherly: There are several issues of concern, but I will take a glance at the following: exposure and software vulnerability.
With regards to exposure, the majority of critical infrastructure devices aren’t connected to the Internet and aren’t subject to malicious online attacks. Unfortunately, a substantial amount of SCADA devices haven’t been properly configured as the research paper by Eireann Leverett has pointed out. And realistically this is a lower-bound on the potentially vulnerable computers, as Shodan at the time was mostly focused at looking for web servers. I suspect that scanning for SCADA-specific protocols, such as Modbus, would reveal a lot more devices.
The developers of SCADA products have a poor history of responding to security advisories by penetration testers. There are numerous incidents of security professionals being ignored repeatedly when contacting SCADA vendors about vulnerabilities in their software.
Cyveillance: The “internet of things” boils down to making everyday items connected to the internet, like one’s refrigerator or other appliances. This new generation of internet-enabled devices is being designed from the beginning with security in mind… no?
Matherly: You would hope so, but that is unlikely to be the case. For example, just a few days ago an exploit was posted that would let anybody control a Samsung TV that’s connected to the Internet This isn’t an isolated incident, and as more of them get connected to the Internet more people will try to find vulnerabilities. Many companies that develop appliances haven’t faced the security threats that the Internet opens them up to. As such, I doubt they will be prepared for the Internet of things that might be coming soon.
Cyveillance: What type of outreach do you offer to help organizations secure their exposed devices? I understand ShodanHQ has been working with some universities…?
Matherly: Yes! For universities and non-profit organizations I provide increased access to Shodan, greater API options and other custom features. I’ve written new filters and created new API plans to help security researchers get what they need out of the data. Often this results in them finding exposed devices, which then forward the Shodan data to the relevant CERT. And system administrators are using Shodan to make sure there aren’t internal systems exposed to the outside world. If you’re a student, professor or work in IT at a university send me an email!
Cyveillance: Does the inevitable increase in the number of systems using IPV6 present any problem to a system like ShodanHQ that visits systems based on their IP address?
Matherly: I foresee slight changes in the Shodan IP selection algorithm to accommodate the increased search space, but the scanning won’t change fundamentally. On the flipside, a lot of new devices will be exposed to the Internet that currently aren’t. I look forward to expanding Shodan to IPv6 and seeing what devices can be found.
Cyveillance: What’s next for ShodanHQ? Are there any new projects or features on the way anytime soon?
Matherly: Lots of stuff! The Shodan crawling software has received a major overhaul recently, and it has let me scale the architecture more effectively as well as add a lot more services to scan. Over the next year, I want to vastly expand the number of services/ software that Shodan indexes. And very importantly, I will begin storing data on ports that are open but don’t return any searchable data. At the moment, every service I scan has to return some form of text that users can search. In the future, it will be possible to find computers simply based on publicly visibile ports.
And I’m also developing a new website that will make it easier to analyze and create reports out of Shodan data. It’s fun to search Shodan and find devices, but it can be challenging sometimes to find exactly what you want. To solve that issue I’m working on a new project that has been designed from the ground up with knowledge of all the data Shodan contains. This means you can browse your search results on a Google Maps-style map, select areas in charts to filter down search results and perform analysis on aggregate search queries.