New codebase, spectre/meltdown and thousands of surprise hosts

It’s done, the new code is live! It took me about two weeks to get it running properly and I saw some weird things a long the way that I’ll share here.

First, the code. Sequentially (“verticallly”) performing all actions per host works nicely. Because the bots do a lot of things in parallel, they still hit the flood detection on the infamous trigger happy firewall of a specific customer. That is now solved by pushing configs to the bots that set limits on the number of simultaneous scans, where possible grouped by subnet. Scaling is then done by simply adding more bots/nodes.

Al lot of technical debt is paid, and I fixed quite a few annoying little bugs. I know I shouldn’t, but I just could not resist to code some new functionality. Call it lack of discipline or maybe necessary for motivation, but the result is that ShadowTrackr now checks all your domains against logs of issued SSL certificates (Certificate Transparency logs). You get notified when a new certificate is issued, and all subdomains are pulled from the logs and checked when you enter a new domainname.

There is a ShadowTrackr test system, but it doesn’t have the scale of the real system. And besides that, there is always unexpected behaviour with real life internet data. When I released the new code in production, the systems took a major performance hit. I found some minor bugs, but CPUs still ran at 100%. Then I found out two problems unrelated to the code were causing this.

The first was Spectre/Meltdown, and after resizing some machines (thank you cloud!) the situation improved. Something still felt off and after some digging I found that one client with a /16 subnet had about 4200 new hosts. These hosts all suddenly appeared and disappeared within just a few days. The hosts had no open ports, no associated urls and didn’t appear in any passive source. Also, the client had not installed 4200 new hosts. In the end we’ve removed them from the systems. Although we’re not sure where this came from, I suspect that some firewall or proxy had a little party and decided to answer a standard subnet SYN scan with RST packets for 4200 non-existent hosts.

The big vertical refactor

At the request of some organisations I added some code to scan servers for a particular problem. These organisations are marked as critical infrastructure, and they often have access to material that is in a responsible disclosure procedure before it’s available to the general public. If you ever have this problem along with PoC code (any language will do), I'm interested :-)

I’m happy to do this of course, but the old flood detection problem reared its ugly head again. This problem often occurs with larger organisations that run multiple servers on the same subnet behind the same firewall. If the firewall has flood detection enabled, a ShadowTrackr node that hits multiple ips in the same subnet behind that firewall will be blocked. The blocking is usually only for 5 minutes, but that’s enough to generate a lot of useless messages on the timeline. It’s a bit like shitposting on Twitter.

So far I’ve done scans horizontally, just like cencys (pdf alert). The solution for flood detection so far was a fancy algorithm that divided all checks and scans over the worker nodes in such a way that no node would ever hit multiple ips in the same subnet range within 5 seconds. 5 seconds are the default flood interval setting found in most big corporate firewalls. When adding more custom scans, this solution isn’t working anymore and the fancy algorithm will become a drag on the database while we’re scaling up too.

I’ve been wanting to refactor some parts of the code to offload more work from the database to the nodes anyway, and this seems like a good time to do it. This is what I’m spending most time on now, together with more vertically based host checks and scans. It’ll take some time to do properly and this means I’ll have put a hold on some other ideas for now. Bugs get priority of course, so please keep sending those.

More sources for news and dataleaks

Your keywords are now used for monitoring more sources in the categories you have already specified. On top of that, we've also added more countries. The US, UK and Netherlands were already available, and Australia, Canada and Germany are new. The newsites we monitor are typically the biggest national newspapers and some local newspaper in big cities.

We've also scoured the internet for more copy-paste sites to monitor. If you're not sure why this is useful, read up on it at haveibeenpwnd.com/pastes. There are sites that claim they will search 90+ datadumpsites for you. That is a bold claim, so we checked it.

A lot of the the 90+ copy-paste sites listed are dead links, have errors or have the domainname for sale. For those that do work there is no API or feed with recent posts available to search them. You can post a message and directly after posting you'll get a url linking to your post. If you don't share this url openly yourself, nobody will know about the post or de data you dumped in it. The 9 remaining sites that were useful are added to ShadowTrackr, but to be honest these are mostly sites to share code. In our experience spicy datadumps are rarely found on specific code sharing sites.

We have been monitoring 10 (from today this is 19) of the bigger pastesites for months now with keywords like "botnet". About 80% of the hits we get are from pastebin.com, which is by far the largest of the dumpsites if you look at both volume and the rate at which new posts appear. It's good to look around for changes in de dumpsite landscape every now and then, but at the moment the effort is unlikely to pay off and we better use the time for other items on the todo list.

If you do miss a newsite or dumpsite that you want us to monitor, or if you want to check if your favorite site is included, please contact support.

Well, that was stupid. According to our site stats there are more visitors from India and Ireland than from Canada. Newssites from those countries are now added too.