Datadump keyword highlighting and context detection

The detection of keywords in datadumps has improved. First of all, there are more lines around the keyword match included in the snippet you see on the timeline. This allows you to better judge if the datadump is something you should worried about or not.

When the snippets get bigger it's easy to lose track of where your matching keyword is. For this reason keywords are now highlighted in the snippets. If there are several keyword matches, the snippet will consists of multiple matches, with a few lines before and after each match.

A bit of an experimental feature is context detection. The data is now scanned for certain characteristics. For instance if a datadump looks like a password dump, you will see an alert icon next to notification. If it looks like the data contains API keys or password, you'll see a warning for this. If things work well, I'll likely do more work on context detection.

-- edit 26 may --
Unfortunately, the context detection worked much better in test than in production. We're seeing too many false positives. It's off now, and back to the drawingboard.

HTTP/2 support and annoying bugs fixed

The number of ips and urls to monitor keeps increasing. This is good news of course, but it also means needing to pay attention to scaling properly. Last week it was finally time to drop a very inefficient JOIN statement from the frontend code. This has been in the making for more than 6 months, since the legacy code and data requiring the JOIN had to be phased out first.

Another performance boost is that since today ShadowTrackr supports HTTP/2. The new ability to process parallel requests helps speed up the loading of some of the slowest pages.

Some clients had a high volume of annoying messages on their timeline, and that should be fixed now. The most important changes:

• Clear and timely SSL certificate messages
• Websites for ignored urls are now ignored too
• The false positive for doubly issued certificates is gone

Lastly, Amazon Cloudfront users experienced some timeline spam due to a lagging ip range update. This resulted in a lot of new ip messages. As of today, Amazon and Cloudflare ip ranges are updated automatically and this problem is fixed.

If you still have annoying messages that you'd like to get rid of, let me know!

Finegrained Microsoft cloud detection

One client rightfully complained that his timeline got all clogged up with not so useful banner changes. I was fixing this by moving the details to the details section (duh) of the notifications. This way they only become visible after you explicitly click something. While working on this I noticed that the most chatty servers in terms of banner changes where Microsoft Exchange online servers. This should not be, cloud services like this are supposed to be recognized and handled differently.

What was happening? I immediately updated the Microsoft IP ranges, which up until now was done manually. But this only fixed part of the problem. There were still servers that were clearly part of the Microsoft cloud that were not recognized. Microsoft publishes the ip ranges for the services (Office 365, Exchange, Sharepoint, Skype, etc,) they offer in these big 5 clouds:

• Worldwide Public and Government Community Cloud (GCC)
• U.S. Government GCC High
• U.S. Government DoD
• Germany
• China (Vianet)

I thought that "Worldwide Public and GCC" had to be located in one place, and since most of the ips in this range where in the U.S., it probably all was in the U.S. Consequently, all these ips were labeled "US public" in ShadowTrackr.

I did also know from some Dutch government clients that their servers are located in the Netherlands in the datacentre MS calls "Europe West" (yes, it's a shame they don't call it "Netherlands Central"). After a bit of searching it turned out that there are many more of these local datacentra. Microsoft even publishes a nice map that shows them all: https://azure.microsoft.com/en-us/global-infrastructure/regions/

And there is a weekly updated XML file available on a different part of the Microsoft site. This XML file has ip ranges that only partly overlap with the big 5, which explains why ShadowTrackr did not recognize some servers as being in the Microsoft cloud. Problem solved!

So, if you have servers at Microsoft you might see them change the cloud description on your timeline. The old cloud clusters will still be available for about a week on your attack surface map in parallel to the new (actual) ones.

Oh, and cloud ip ranges are now automatically updated daily :-)