Major Update

I like to do small incremental changes, test them and put them in production. This is less risky and allows me to focus. Unfortunately, not all changes can be done that way. It was time for a Major Update.

There were some performance tweaks that I wanted to push and I had a solution for a long standing queueing problem. Both required updating the internal data structure and migrating the data, which was done during the weekend.

Some of you might have noted that the number of hosts found kept increasing in the last weeks. Old hosts were indeed not always properly removed, but the big issue here were clouds. If a DNS A record was pointing to Microsoft Exchange Online or Amazon AWS, then the IP returned kept changing. After a while, you'd see a whole group of IPs around the url in the attack graph. ShadowTrackr now recognizes clouds and replaces the group with one black dot, with the name of the cloud next to it. The result is that if you are a cloud user, you'll have less assets now.

Another problem, mostly for the bigger clients, was the clutter on the timeline. If you're scanning the internet, you will encounter servers behaving badly and a lot of weird, unpredictable events. Some modules, the DNS module in particular, didn't handle this properly. This is now fixed.

To further cleanup the timeline, some less interesting messages (like a change in the servername) are no longer visible on the general timeline. They are still there when you need them for a more thorough analysis, but only on the timeline of the asset itself. The timestamps no show the timezone (UTC), the less useful source information is left out, and the messages themselves better explain what is happening.

Also, some new features have begun to slip in (it's just to hard to resist temptation). ShadowTrackr has started gathering BGP prefixes to build up data I need later on, and if you run an FTP server you'll noticed the security settings are checked and the banner is grabbed. There will be more of this in the coming weeks :-)

As with any Major Update, you're always afraid his evil twin Major Error comes along. The weekend went well and so far it's only been minor bugs. I expect there will be some new bugs in the coming period. Please let me know if you find one.

Asset grouping and deleting urls

About 30% of the clients have hundreds of urls in their assets, and some even go beyond 1000. This very long, flat list is not very user friendly. So, time for some UX improvement.

I took a look at several of these long lists, and, as expected, there are many subdomains for the same pay level domains. This is good, since it allows grouping them. The new view under assets lists all your domains as clickable groups with the number of subdomains in front:

+ (3) domain.com

When you click it, it will show all subdomains:

- (3) domain.com
  a.domain.com
  b.domain.com
  c.domain.com

Much better :-) Just as before, the domains are sorted alpabetically on the pay level domain ("b.a-domain.com" will appear before "a.b-domain.com").

Another change is that you can now delete all urls, not just the ones you have added manually. I'm still experimenting with how this should be done, and it's likely to change again someday. There are urls that you legitimately want to delete (since they're no longer yours for instance), so the option should be there.

The problem is that some urls are related to you, and even if you don't like it, they will be discovered and added again. No matter how often you delete them. A delete option for these offers false hope, and I don't like the disappointed that follows later on.

Also, I think it's a good idea to keep monitoring your old urls that expired. These are the ideal candidates for setting up phishing sites. The same holds for those internal urls that should not appear on the internet. I hopw to come up with a better solution one day, but until then: be careful when deleting!

The blacklist counter from hell

The blacklisted page up until now listed all your hits on blacklists. That is, every hit is a separate entry in the table on your screen and is counted as a problem. The number of problems you have screams at you as a number in a red dot in the menu on your screen. Sounds good right?

As one client showed me, some ideas sound good in theory but turn into the blacklist counter from hell in practice. ShadowTrackr at this moment checks your ip addresses and websites against 127 blacklists. A lot of these blacklists overlap and from a security point of view that's just fine. You'd rather be notified twice than not at all.

When an ip gets listed as a source of SPAM on one blacklist, the chances are high that a couple of other blacklists will pick it up too. Since the counter counted the number of blacklist entries, 2 machines getting listed on 4 spamlists resulted in the number 8 screaming at you from the bright red dot. That is not the user experience I intended. In that case you have 2 problems, not 8. The counter is fixed now, and all blacklist entries are sorted per asset now.

What remains is the question on how to handle notifications. For the first time your asset is listed on any blacklist, everyone will want to receive a notification. But how about the second or third blacklist that same asset gets listed on? Do you want to know? I myself would like to get notified of every extra blacklist an asset appears on, so I left it on for now. But if enough users convince me otherwise I'll be happy to turn it off of course. Just let me know!