Bad bot

My web sites don’t get a lot of traffic. One of them dealing with the birther movement was getting 40,000 unique visitors a month, but after I stopped publishing new articles, that dropped sharply. What continued was search engine spiders (bots) crawling and indexing the pages. At least 90% of my traffic is from search engines.

One of the bots that tore up one of my sites, accessing the same page over and over a thousand times, was SemRushbot. The amount of web traffic it generated was staggering and I filed a complaint with them for damages. Because of that abusive bot, I’ve been watching the spider traffic more closely and identified another bot that is spending a lot of time on my website, DotBot.

Neither of the two bots is a search engine. One supposedly monitors ad campaigns for a site’s competitors, and the other has to do with eCommerce. None of my sites has ad campaigns or any kind of eCommerce. Those bots have no business on my sites.

The standard way to stop a bot is to ask it nicely to go away. That’s done with the robots.txt file. The problem with that approach is that the spider can just ignore the file and crawl your site anyway or it may take some time for the spider to find out that you’ve changed the file.

In the case of SemRushbot, it appears that it does respect robots.txt because in the last 24 hours on the site where that bot caused so much trouble I found that it had accessed the robots.txt file 13 times, sometimes twice in the same minute, but that is the only file it accessed. DotBot is not so cooperative. It accessed the robots.txt 30 times, but ignored it and accessed 233 other pages–it didn’t get them though. I use the WordFence plugin on all my sites and one of its features is the advanced blocking capability of banning a user agent. All the DotBot traffic was rejected with an error code. Another bot that spends a lot of time on my site and provides no value is AhrefsBot, and I block it too.

The most prolific bot on my server right now is BingBot for the Bing search engine. That’s fine because I want people to be able to find my site if they want to. GoogleBot is there about half as much.

On my largest site I have added the location of my sitemaps.xml file in the robots.txt file. That contains the date the posts were last updated and hopefully the spiders will be smart enough not to re-scan pages that haven’t been updated.

Posted in Bad Commerce, Technology | Leave a comment

Web attacks

All of my blogs were down yesterday and I spent several hours scrambling to get things back up.

Someone started attacking the Contact Us page on one of my other blogs. I got over 76,000 spam emails from it, but the larger problem was that the page accesses were coming so fast that they blocked any legitimate traffic to my web sites (they’re all hosted under the same account). My web hosting company,, throttles the number of emails the site can send, but that number wasn’t enough to keep me from being inundated by spam, and it didn’t take any load off the site.

It wasn’t just one computer doing this. Analysis showed over 500 different IP addresses participating in the attack. (All of the ones I checked were from China.) Software on my sites blocks excessive accesses by a user, but this was hundreds of them, individually not over the limit, but collectively devastating.

To fix this, I had to put the site offline using tools outside the normal web interface. I deleted the contact form, and put a deny rule in place so that the web server won’t even pass a request for that page to my site. I replaced the contact form on all my sites with one protected by reCAPTCHA so they won’t appear ripe for abuse.

The analysis and repair was a huge effort, and the sites were down for half a day before I was even aware of it. Someone with less experience than I would probably have had to pay a consultant to fix things, and it might have taken days. As it was, my web hosting account was 40 minutes away from being disabled because of all the spam. My point is that these attacks are a big problem, and one that needs to be fixed at a higher level than the individual blogger like me.

[Update]: It happened again, only worse. Starting around the end of December, 2017, I was hit by a hotlinking attack. A web page embedded images from several sites including one of mine and then started accessing that page. In fact over 4,000 different IP addresses accessed that page, some as many as 17,000 times. I found that the IP addresses were from Amazon Web Services across the world that hosted a site uptime testing service.) The result was that I ran out of bandwidth and my host,, shut all my sites down. This resulted in a multi-day outage because my hosting plan was frozen until I upgraded the account to pay for more bandwidth. In this instance the ultimate solution was to block hotlinking altogether.

As part of the solution, I moved all my sites to another hosting company that doesn’t have a bandwidth limit on the account.

Posted in Technology | Tagged , , , , | Leave a comment

Can I guess your password?

I probably could. Just give me a few billion guesses. Web sites that you log into don’t store your password; they use some cryptographic technique to store a scrambled version of it. When you log in, they scramble the password you enter the same way and compare the scrambled versions. If they match, you’re in.

The problem is lots of those scrambled passwords aren’t stored using the best technology available, and there are huge passwords lists available on the Internet so that someone can start guessing. Even an 8-character password can be guessed through brute force.

Good passwords are nigh impossible to remember, and it’s tough to keep it all straight. What some folks do (myself included sometimes) is to use one password for multiple web sites and services. That means that if your weak scrambled password gets out, then every log in using that same password is now open for identity thieves, email spammers and other malicious actors.

There are a couple of solutions. One of them is to use a scheme whereby you make up passwords that you can remember. Security expert Bruce Schneier has one of those. I use something like Schneier’s approach for my Facebook password and a couple of others, but the best approach for me is to use a password manager. There are several around. These programs use strong encryption to store your passwords, and they can create strong random passwords for you to use. I use one called KeePass, but there are others. One nice thing about KeePass is that you can put the software on a flash drive or a cloud drive, and run it from there. It’s not necessary to install the program on your computer. There are versions for Windows, iOS and Android.

It seems like there are security breaches and hacks happening with increasing frequency. More and more of our data goes online. I do what I can to keep it secure.

Posted in Technology | Tagged , | Leave a comment

I feel like I’m being followed

I posted a message last year on the Tesla forum about an upcoming round trip of about 200 miles and whether I could make it on my then Model S 60 in Winter. The consensus was that I’d have to find some charging solution at my destination. Shortly before the trip, Tesla opened its Supercharger in my destination of Columbia, SC.

My next challenge was a round trip to Birmingham, AL. Again, I was going to have to find some local charging solution that was going to be very inconvenient. But a month before the trip, Tesla opened its Supercharger in Birmingham.

So a couple weeks ago, I was visiting family in Martinsburg, WV. Now with my Model S upgraded to 75 kWh, this round trip was fairly easy, but still if I drove a lot at my destination I’d have to find some charging. When I got there, I was greeted with “Oh, did you know they’ve built a new Supercharger here in Martinsburg?” It’s not on Tesla’s map, but the station is physically complete (photo following), but wasn’t online. I’ll be back there often.

Now I have several likely trips coming up to Charlottesville, VA. Last time I went there, charging was messy. I had to use some 120 V AC charging, and then for an unexpected side trip I had a 20 minute drive to get to Martin Horn Inc. to charge at his solar-powered HPWC, and a long sit in the car. On another trip there I had a long sit in the car at a Nissan dealership in the middle of the night. Guess what Supercharger is under construction now?

[Update: 9/23/2017] The Charlottesville Supercharger is complete, and I charged there this week.

Posted in Electric vehicles, Travel | Tagged , | Leave a comment

Habitat tablet

So I was out on the Habitat build site in Greenville today, using my Habitat tablet computer. As you can see from the photo, it sports a 7” diagonal screen and has a convenient kick stand. You can write on it with the optional stylus, shown.

It is extremely useful for database applications, such as storing dimensions for construction. It can be used with windows.

Photo of Scrap piece of drip edge with carpenter's pencil

Check out some of my other Habitat photos.

Posted in Habitat, Homor | Tagged | Leave a comment

Don’t order online from Pizza Inn

I get email offers from Pizza Inn from time to time, and some of the coupons are pretty good. Today I clicked to order online and the order for two pizzas was $3 more than what it said on the coupon. I thought it was an online glitch and that the store would charge me the amount on the coupon.

Wrong. There is a hidden charge of $1.50 per item ordered that they tack onto the order. Nowhere do they disclose the charge that I can find, nor is it detailed when you print out order. All it says is that your $8.99 pizza costs $10.49. In fact, the Terms and Conditions explicitly says that there are no fees for ordering online

6. Fees Schedule

QuikOrder does not add any fees or costs to your order when you place an order or use this site.

I can’t blame this on the restaurant because the web site added the $1.50 fee—there was no discrepancy between the online price and what the restaurant charged.

To add insult to injury, when I got home, what I got wasn’t what I ordered (too spicy pepperoni instead of pineapple). I was too bothered by the overcharge at the store to check the order carefully.

I call it fraud.

Posted in Bad Commerce | Tagged | Leave a comment

My new car

What? You only bought a new car in September!

I upgraded my Model S 60 to a Model S 75 “over the air.” I paid some money and my old car turned into a different model. It took about 10 minutes. The car’s range increased by 39 miles, and the top speed increased from 130 to 140.

Posted in Electric vehicles | Tagged | Leave a comment