Predicting auto insurance claims with deep learning

Part of the fun of learning data science is seeing how quickly it can relate to your usual roles and responsibilities. While AIG sells insurance, I catch criminals. While underwriters make predictions, I protect data. So when I needed a testbed to practice deep learning and better understand a business perspective, I turned to Kaggle’s Porto Seguro’s Safe Driver Competition.

This competition is fun because you are asked to “… build a model that predicts the probability that a driver will initiate an auto insurance claim in the next year.” To save you time, you should know our team did not win, and the 1st place winner’s submission is a fantastic read. However, I did want to cover some of the critical lessons I learned.

The cohort and building a team

Currently, Jeremy Howard is teaching the part 1 to a group of local and international students. A 7-week course it covers many of the fundamentals of setting up and running a model to drive results. Practicality first, technicalities second, taking the free classes were so good I had to apply again when he switched from Keras to PyTorch. Many of the cohorts have some fantastic articles (here, here, here) based on what we are learning from class.

For me, competing head-to-head against other data scientists helps solidify my learning, and so once again I turned to Kaggle. Since I had previously worked on time-series predictions for web traffic, I found the Porto competition especially tempting.

A great thing about the cohort is that you can quickly find someone who is also interested in a similar project. Fortunately, I was able to team up with Devan Govender, another student participating.

Due to time limitations. We only had about 8 days to work on the project. This accelerated timeline was great because it forced us to move into the project quickly.

Sharing Data

I have to admit that my Github skills are lacking. Due to the Kaggle competition rules Devan and me set-up a private Github instance to share information back and forth. At the time, there was not a way to install the repo with pip through the Kaggle interface.

Some things I enjoyed about a private GitHub instance continues to be the ease of sharing ideas back and forth. It took only minutes to be able to run what Devan had uploaded. Plus there were more than a couple of times that GitHub provided a way to get back to a known good state.

A few commands became my bread and butter for using GitHub.

Clone- gets a copy of the project I am working on

Status and Pull- We can see that after the clone command, we have the folder with the GitHub code. Additionally, we can check status (it is up to date) and try a pull (again it is up to date). Extremely important before we start making changes to the code.

Push- After we have made our changes we label the modifications that we fill in a commit about the changes made and the files changed. Then we push it back to GitHub for someone else to use.

Additionally, in the Jupyter notebook, I set-up the code so that we would not have to change too many things do to pathing.

Also, now that the competitions are over, we can release the code allowing everyone to see it in its unfiltered madness.

Getting data in the right place

Unlike other competitions, Porto’s data is anonymized more than I would expect. The data columns labels have nondescript categories, but at least the columns are labeled into the types of data such as continuous, categorical and boolean.

Incorrect features can be a real problem for records and to correctly use them. I can barely understand what they are trying to do here. It is much more difficult to go back and interpret the meaning of the values provided.

Fortunately, we can work through it. For example, there could be a category with a value of 1 in it could be interpreted in the following way:

  • a boolean value: True the insured car was in an accident
  • As a categorical value: The insured vehicle is a Ford
  • Continuous value: This car has gone 1 mile

However, what happens if the next record was 3 and how would that describe the relationship?

  • A boolean value would not make sense because booleans are only yes or no.
  • A categorical value would suggest the car is a Ferrari, not a Ford. This alteration of models could drastically change the chance of a claim.
  • It could be continuous, but the difference between a car with 1 vs. 3 is likely insignificant.

As you can see accidentally mislabeling the value can have a significant effect on the data.

Luckily, in this competition, we were told which value categories. However, I wanted to doublecheck them. So I ran some analytics

  • A boolean should at most have 3 values (True, False, NaN)
  • A category column will likely be in the double, but not triple digits.
  • A continuous will have many many unique values. Going back to the mileage example, imagine all the different mileage counts that would be available. Almost every car would have a unique category! One for cars with 1 miles, 2 miles, 3 miles… etc

We can see below that the cats, bools, and conts all make lots of sense. At least we are not as blind as we were before.

Boolean and categories and continuous oh my…

The most significant oversite I missed was how many variables were missing and how to solve for them (some categories had over 50% missing). We see these as nan values, represented as a -1, in the code. Now depending on the type of the data there can

  • Boolean and categories can easily just add the nan as an additional category. Not having a value can sometimes give just as much information as having it.
  • The continuous variables are a little tricky. Leaving them with a default of -1 can be goofy. Assuming that the model rated low mileage vehicles favorably, any car missing a value would be rated more favorably than brand new cars! What we tried to do late minute was just take the average of the other values to ensure that it did not impact the prediction.

I think these methods worked out fine, but it goes to point out the difficulty in working with data anonymized in this manner.

Reverse engineering features

I had major problems with training my data. The first thing I tried to do was to properly go back and classify the number of unique values in each category. This check helped ensure the data was correctly labeled or could not be improved. Even with the alterations, we continued to have problems.

Getting the right learning rate seemed fickle and when we were using the Gini coefficient, it took some time to move downwards. There were just too many things to calculate.

At this point, we saw that most competitors dropped the less important features. The last-ditch effort was attempting to remove as many as I could to better understand what might have going wrong. It did not help much.


When the dust settled, we placed in the Top 16% of over 5,000 submissions. I certainly learned lots about how to coordinate better with my partner (thanks Devan!) and the importance of understanding the different types of features. However, more it helped me realize how important it is when collecting data to know what it looks like and how to represent data. We are still long ways off from just throwing all our data into a black box to see what magic pops out the other side.

Disclaimer: Although I chose to work on a competition hosted by an insurance company, there is no overlap between my hobby of data science research and my responsibilities at AIG. Only personal computing resources, personal free time, and competition provided data was used. 


9-months in the “hobby” of deep learning

Deep learning, AI, machine learning, and all of those others buzzwords are spouting out everywhere. No domain is safe from marketers trying to use these terms to sell a product, and no startup would be caught dead without them (or blockchain). So to enact my due diligence I wanted to jump on the deep learning bandwagon. The problem was my plate has been very full. These past 9-months had:

  • Daddy duties
  • Husband duties
  • Work duties
  • A month of Executive courses
  • Preparing for DEFCON
  • Fighting off a hurricane Harvey

So can a professional just take up deep learning as a hobby? Sure can!

I had tried going through Andrew Ng’s Coursera course, but I quickly got sidetracked. Fortunately, I discovered Fast.AI (Jeremy Howard and Rachel Thomas) and launched myself through the first two modules. I even got accepted into their follow-up part 1v2 as an international fellow. Despite all the time factors, there is something addictive about getting my hands dirty running and altering the scripts.

The Rig and the joy of GPUs for Deep Learning

Some people love their cars or their guitars, but I am passionate about my computers. Lesson 1 form Jeremy is setting up an AWS instance. While the AWS instance worked, I quickly decided I need to take advantage of a GTX 1080 that sat idle most days.

There are several links (herehere, and here) describing the best way to make a deep learning rig. Fortunately, it is mainly just a gaming machine putting on adult clothing, and it only took a little bit of tweaking to get scripts running. The most significant change I needed was buying an SSD to hold my training data on and install a new version of Ubuntu.

Setting up SSH to allow me to log in remotely has also been vital. Every morning I can spend about an hour drinking my coffee and getting ready for the day to start. Having a remote connection to my rig allows me to quickly pick up exactly where I left off and not need to carry around the machine with me. Indeed, my deep learning computer does not even have a monitor because I merely login remotely, even at home.

How have I not used GitHub?

I have known about GitHub for quite some time, but I have not routinely used it. These last few months I have gone from one project with 9 lines of code to about nine notebooks. I feel like I am barely scratching the surface. I pull, push and clone but there is so much more I have not got to yet. It does allow me to quickly update and share my updates which I find valuable in case my rig went up in flames.

The Jupyter notebook

It was also shocking how much I have grown to love using the Jupyter notebooks. All the documentation, and saved outputs readily repeatable. Troubleshooting is amazingly more comfortable for me, and a large part of the data is just making sure I accurately see the formats for it. Jupyter gives that to me in an easy to understand way. I wish I used it back with Kali for pentesting documentation so that everything is both rapidly reproducible and documented.

The best features when dealing with larger training sets were the timing features for individual blocks. Having the ability to see how long an iteration takes and having a verbal warning when something completes is very valuable. If data is taking 30 minutes to load, finding an alternative loading mechanism makes much more sense until you need it.

Are you solving problems?

If you are expecting to solve world hunger, we might be a ways off. However, an excellent standby for testing what you have learned is with Kaggle competitions. The course has plenty of real-world problems with real-world data. Seeing what other groups are solving has been helping me think about what I can apply to work immediately. Not with a billion dollar budget, but with what I have right now. Here are my three favorites.

Cats and Dogs — Kaggle Competition

Everyone needs to figure out how to better identify cats and dogs. This contest goes out of its way to keep the fight alive. Using several pre-computed models users can predict if an image is a cat or dog. In my 2000 images, results in only 13 incorrect answers. Here are some random examples of the correct pictures.

So that is pretty good, and these predictions all makes sense. However, when we look at the incorrect cats, we see the following.

We can see why a computer might get these wrong. These are bad pictures. My two year old wouldn’t get these either. The beauty of the project draws from its simplicity and ease of understanding. A great first project.

Statefarm- Kaggle Competition

I find this one much more interesting since it classifies human behavior. The images are cut up into multiple categories trying to show that humans are doing silly things while driving. While there are several defined activities, it is fun to catch people being a jerk in many different ways. Most of the distracted driving is simple.

Is the driver distracted and if so how? Are they texting, yelling at someone on their phone, drinking a soda, or something else? These classifications are extremely easy for a person to describe. However, it has been only recently that you can start thinking about how to get a machine to learn them.

As an aside, you can really mess with the dataset for this one. If you decide to have drivers in the training and validation set, you can add bias to the models. For whatever reason, my first interactions incorrectly labeled drivers with glasses as distracted. Every. Time. Upon review, I discovered my all my unsafe class predictions for a particular category had glasses! Imagine an employee sending a dataset to production that made the same mistake with hair length or skin color. It is terrifying.

On the plus side, I feel that this could have the most profound impact on behavior. Imagine people calling out when they are slouching in chairs. Alternatively, if a child is climbing on something that is too dangerous. Alerts, warning, corrections can help keep people safe and drive changes in behavior to make us better and safer.

Web Traffic Time Series Forecasting- Kaggle Competition

I go into this competition more in-depth here. However, since posting the contest has ended and I placed in the top 30%, this is pretty high up there considering it was the first competition I went into alone and unafraid.

The misunderstanding I had here (cat vs. cont variables) I was able to work on in my 2nd competition. So I am still thrilled about placement. Additionally, it started off my deep love for non-picture type problems. Plus there are two other similar examples right now which I will likely compete in.

My next steps

In case you want to look at some unfinished code you can check out my work here on GitHub. However, I would recommend that instead you go and start taking a look Jeremy’s course to get into deep learning. Seriously, take a day off work and try this. I know I learned mountains from it and think the way he approaches the teaching of others fantastic.

Its not THAT hard.


Friday the 13 “spooky” leadership lessons for October

Living in Texas, I do miss out on some of the delightful aspects of October from childhood. The crinkle of leaves, burning firewood, and the brisk night air is replaced by just… moist hot. However, I can still sneak away from being an adult to rewatch all my favorite horror movies to enjoy the season. So to stay in the holiday spirit here are 13 leadership lessons on Friday.

1. The most significant challenge is never really expected

None of the characters go into a story expecting machetes, hatches, or claws. They have entirely different concerns and goals. Looking back at my most significant challenges every single one seemed to come from nowhere as I was more concerned with other things. Running through bizarre scenarios is a very beneficial tool for rapidly dispatching mundane challenges.

2. Don’t split up

Perhaps the cardinal rule. Tempting because a team can cover much more ground but a loner is more likely to get stuck in something way over their head. Try to keep at least two people on essential projects to support each other. Going it alone? You might lose them and much more.

3. The people in charge need incredibly compelling evidence

One of the most prominent tropes in horror is that the parents and police don’t listen. While it makes sense for a moody teenager label others dumb, we all understand that an immortal unkillable machine is quite an extraordinary claim. The turning point is always when compelling evidence (usually a body count or seeing the monster). Always use strong proof when appealing to stakeholders to make a decision.

4. Resourcefulness is key; Silver bullets fail

Unfortunately, silver bullets are a one-stop solution only in movies. Even when used they rarely have the intended effect, and the heroes resort to their incredible resourcefulness with the tools and supplies they have. Often jerry-rigging something to help hold the monster at bay or make a daring escape. Just skip the bullet. Use what you have around you instead of trying to get something with impossible requirements.

5. Learning is critical

Always be willing to learn from your mistakes. Predictably approaching a problem because that’s how it was done in the past can leave your team open to some harsh realities in a changing environment.

6. Somebody always warns of the impending danger

There always seems to be warning signs of impending doom that seem apparent in hindsight. News reports about increases in phishing attacks during the holidays, the release of new exciting codes, or even grandma talking about her friend losing her retirement to fraud. We can’t dwell on everything but consider that there is some underlying truth in statements.

7. Your competition is evolving, so should you

Nothing will continue to work forever. If there is no innovation, you will be leading a group forward to the future but back into the zombie-filled catacombs. Without making changes to explore and exploit new opportunities the team will stagnate.

8. Simple is good

The most complicated plans can be put to shame if everything doesn’t line up. Sometimes the most straightforward solutions are the most effective.

9. Take care of yourself

You can’t show leadership if you are out of action. Overworking a body is almost as silly as running away from an ax murderer in heels. Eat healthy food, exercise, spend time with loved ones and on hobbies. Make sure you are ready for work each day.

10. Watch your hubris

More humanistic monsters often make broad generalizing statements condoning their mayhem. A solution that makes sense for one variation has a danger of being applied too broadly.

11. You find out your real companions when everything goes wrong

If team members are suddenly disappearing to leave you to face the problem alone, they will probably run away from every challenge. Not physically but through a deluge of excuses and rationalizations. It is best to sort out who will have the courage to face problems head-on and help the team despite the circumstances.

12. Understand whats behind the mask

Masks are used to portray what we want others to see. We all present a side of ourselves when we go to work, and we often miss unique aspects of our peers. Try to peel back the mask a little bit to see hidden talents or motivations. You can find a secret rockstar passionate about a direction you have never considered.

13. The problem never dies

You can’t just kill a problem. Sure you can slow it down and incapacitate it, but when you turn your back, it will rise again. There are always going to be new iterations of the problem that keeps popping up and evolving. The most important defense is to learn and apply what you know in the past so that you can better deal with the problem next time.

Hope everyone had a few moments to think about how their favorite horror stories and how those lessons, as fantastical as they are, can provide a tool for your leadership toolbox!

Have a Happy Halloween!


The blunders from my first data science competition that you can avoid

There is a dark room in my house that has no windows, is 5 degrees warmer, is a large part of my electrical bill, and has a high pitched whirring. My computer lab attests to my addiction for overclocking computer components. While it started as a testament to cryptocurrencies, this time my GPUs have been overworked trying to finish the last few epochs of my models for the Kaggle Web Traffic Time Series forecasting.

Kaggle is a free online platform that allows users to learn how data science works and compete to win different recognition and prizes. I used this platform as a way to test what I have been learning for the past several months. I landed on the Web Traffic competition that was due to end in a couple of weeks. At one point, a thousand teams were trying to estimate Wikipedia’s page views from Sept until Nov in an attempt to win a $25K prize. It was tough and unfortunately, and while I learned lots, here are some of the major mistakes I ran into along the way.

Educational resources and thanks up front

The only reason I got this far was thanks to the Fast.AI MOOC taught by Jeremy Howard that I have been studying for the past several months. While my code for the competition is here, I would suggest you borrow Jeremy’s Rossman code for a better understanding of building out time series problems.

What are we trying to predict here?

Being given a large stack of historical data we are working to predict Wikipedia page visits per day in the future. Best advice is always to split data into three different sets.

  1. Training Data- school book of examples
  2. Validation Data- practice problems with answer
  3. Predictions (testing) – the teacher’s test

The competition gives you the training data which you can split up into the different sets. While there are many different ways to run the models, I was using Keras to make my predictions. My model runs appeared like this.

Let me decrypt this for a second. Epoch states that the model will run through the calculations one time. ETA has time remaining (most iterations are 11 minutes), and the loss (the degree it was wrong) sits at 26. My training data set had 49.7 million, and my validation data had 21 million points.

Ideally, the training set would be even larger than the validation data (the goal being around 80-90%) however, the 8 GB on my GPU could not hold any more data in memory. Which brings me to my first mistake.

I was not able to use multiple GPUs

Although I have several GPUs at my disposal, for some reason, I could not get the data to spread across multiple devices. Nor could I run different models assigned to individual models. Having hardware that was underutilized was a huge disappointment because I would have been able to run more epochs and hold larger datasets. Additionally, I opted to reduce the number of features in my data to favor having more days of data. To put it another way, my model did not spend enough time running through its studies.

Here is an example of some good solid training. See how the orange line (actual values from the valid set) closely maps to the blue line (predictions made of the valid sets). This data would have a lower loss rate and do an excellent job of predicting how many visits a page will see.

However, here are some less trained models. I include the red line is the mean for the data since that was a popular method of estimation in the competition.

The most dreaded graph is this monstrosity.

The top models had more runs against it, as it was smaller and much faster to run. The lower three charts do not have the same numbers of runs as the extra data slowed it down too much. So these models did not have around 50 rounds run of them but around 10. Fewer iterations of the model almost directly related to a decrease in accuracy for my data. However, with all the variables it took nearly one hr to run, and I ran out of time because….

The dreaded 12:30 AM data extraction error

Be careful with your data. I had spent my lunch hour setting up a run that lasted 9 hours only to discover that some data was amiss. Most of my data featured several features used to predict the number of visits and would look like this.

When sorted according to date I saw several dates that occurred before the competition began. While searching data I realized that I was taking out the wrong date. In some cases, I was grabbing the first date instead of the second date, which meant my model was considering all the dates to occur on the same day (an example below). This oversight caused errors for 10,000 of my data points.

By the time I fixed the model I had let it run overnight getting a solid ten epochs into the dataset. I finally submitted the model right before the deadline with a sigh of relief.

Interesting details of the competition

One pleasant surprise is how friendly the different participants are. There are many conversations regarding techniques people are working on, trying out, and their thoughts. Even if you are a beginner, some great models take no hardware. A rather popular example was the 1-line solution which gives a real straight forward way to predict the data. I learned lots just from studying this line!

Additionally, although the competition submission deadline has passed, actual scoring will go on for the next few months. So while I sit rather low right now, I am hoping to see my position slowly grow as they update the scores over the next few weeks.


Thanks Interns! Three management lessons as temp workers transition


Businesses everywhere are seeing a large supply of cheap expendable labor depart as interns have headed back to school. Often the butt of jokes, undervalued interns are criticized for being inexperienced, undertrained, and very temporary. However, my personal experiences with interns have shown that they provide valuable contributions to the company if the manager is mindful of a few things.

The Interns value to the company

Interns provide the invaluable gift of work hours. Not free, but cheap. Interns are the most straightforward answer to getting items complete that you just need additional employees for tasks in some form of neglect. An influx on work hours can provide the momentum to push that project past the small hurdles and goals to something sustainable. With the right manager, a good intern can provide a fresh perspective and have a desire to complete their projects before they depart.

However, as with all cheap labor, there is often a trap of providing just additional “busy work” which can be spent or projects that just keep the interns producing something, anything besides just breathing. Instead of working efficiently they might be asked to continue a long drawn out procedure.

Equally wasteful, is putting interns on side-projects that are not important enough for your full-time employees. If it’s not important enough for a full-time employee, then my team is not doing it. It’s not going to be a burden for an intern.

In an unfortunate situation involving a bad intern, you can still get some value by pulling other work off your more productive employees. Don’t throw too much valuable time after the bad.

Management responsibilities to them

There are many diverse reasons why an intern would want to come and work with a company. Future career prospects, the type of work at a company, and hopefully the company’s reputation for running an excellent internship program but I never know what drove them until I ask them. It is one of the first things I should be asking when they show up.

What are they expecting and how can you help them get that.

On this latest batch, I mistakenly fell into the trap of being “too busy” and forgot to complete this step. While we were able to provide many learning experiences and let them provide tangible impact to the business, I might have been able to do a better job at aligning work with their interests if I had not slipped up.

Immediate feedback is also a key component. Professionals write scores about how feedback can be uncomfortable for both parties. I find keeping the tone straightforward and prompting leading questions for improvement helped us finish projects better. Also, the intern doesn’t revert just back to receive mode. These conversations should be modeled more like ping pong, both parties should be speaking.

I’m not the best at stepping back and allowing the process to occur. Often I just want to jump forward and drive. Like most people I know, we feel we are good at driving tasks, and we want to get there faster. However, when I allow myself to get trapped in directing instead of questioning, the results are not as good, I kill innovation, and underserve the intern by thinking for them.

I am also somewhat selfish about my interns succeeding in the program. These are people who have been vetted and groomed by the company and have a large potential for future growth. Having a good network of new up and comers is a future investment in myself and my career. One day, I will need either the intern or someone they know to help out with a project or idea.

The more knowledge and experience I provide to the trainees.
The more I support their pursuit of goals.
The more I will be able to draw from them in the future.

Management and Leadership Testbed

During my one on one sessions with employees upward mobility is a top concern. (If it is not you have other concerns). There is no question that the largest resume builders are high visibility pet projects of management and interns are a close 2nd. Interns allow my full-time employees that trial run in leadership.

The largest misconception about the military is being stuck with a Drill Sergeant barking and spitting in your face 24/7. That can’t be further from the truth. My Marine Corp “internship” was marching around with an infantry platoon. I saw that the marines were teaching leadership from the top officer to the lowest ranking enlisted. My manager was a lance corporal with two months experience making sure I didn’t mess up. He was the one grooming me. That decentralized leadership and autonomy being taught all the way to the ground is a core competitive advantage that both Sailors and Marines share.

In my biased opinion, you should follow this model.

While the military has a constant flow of people moving in and out on rotations, my corporate team doesn’t get that luxury. The more junior analysts do not have anyone to train or practice leadership with on a rotating basis.

Interns solve this.

Suddenly, there is a new, inexperienced team member ripe for training. The influx of temporary employees, allows a manager to put even those junior analysts, in a role that requires the management of the intern. It’s a fantastic testbed for your full-time employees to learn to teach. After all, the worst thing that could happen is the teaching of bad habits, which leave after summer! Even a complete failure, will inform an employee which of their leadership tools were more or less effective.

Ready for the next batch?

Sometimes interns are viewed as a bother, someone to babysit during the summer as you move through your typical workweek. Although I understand the concerns about their limited experience and short tenure, I have also grown to view them as an essential part of our growing and developing m team and would urge you to seek interns out the best you can.


My 3 favorite unofficial DefCon 25 badges

While DefCon has been known to have interesting conference badges, the 25th iteration had an unexpected explosion of intriguing unofficial electronic neck swag. The hunting for and gathering of coveted badges has become a new tradition and this year’s #badgelife built on that tradition. While unforeseen circumstances caused this year’s official badges to be rushed into production, attendees did have a nostalgic combination of throwback badges paying homage to conferences of the past. Fortunately, attendees had many choices to display custom badges that bling, communicate, and even fight from unofficial sources. Often these badges have secret competitions and groups to teach people how to deconstruct and find hidden achievements in their hardware. Although I was far from getting all of these unofficial badges at DefCon, there were three that caught my eye.

1. AND!XOR’s Bender badge

My favorite badge! Last year I fell in love with my little Bender badge after being a winner of the grand elevator rush of DC24. This year’s badge was a huge step up, and it features a full-color LCD screen, a host of LEDs and my favorite character from Futurama mixed with the cult classic Fear and Loathing in Las Vegas. This new badge was a huge step up from last year. The Bender badge has a host of unlocks available to get additional characters, screensavers and a wireless module to interact with other badge owners. They are also cross compatible with many other badges from the regional DefCon groups like DC801. If two compatible badges were near each other, they would flash each other’s logos back and forth between screens. How freaking cool!

A much more well-known feature on the badge was the “Botnet” which allowed badges to fight each other as you develop exploits, patch your badge’s services, and launch attacks. In particular, a successful attack would render the victim badge temporarily unusable as Clippy, BSOD, or a Rickrolling took over for a minute. Suddenly, badge owners were in a race condition with each person trying to hack the other guy first. The loser’s badge sadly broadcasting their shame. The truly devious would launch another attack as soon as the victim cleared the first one.

One hidden feature of the badge is an actual botnet feature that allows the AND!XOR creators to propagate commands across the badges. For example, maybe AND!XOR wanted to start off a Hypno-toad dance party or maybe Rickroll a room. The problem was that DC801 took advantage of this “feature” to hijack the command and control architecture. They were able to infect one badge, which would wirelessly reach out to attack another’s within range and so on. This cascading virus is exciting because there is an IOT mesh net architecture that a virus happily hopped along. Suddenly badges are attacked just by walking through the area! Even after reboot badges just started another iteration of the Matt Damon video clip disabling the user interface for a minute. I am seriously sick of him spinning around. Throughout the weekend AND!XOR and other groups dueled for the control of the botnet and our badges. Fortunately, this seems to have cleared as I got home.

Just take a minute to contemplate this. While users were busy trying to attack each other on an individual level, AND!XOR and DC801 were fighting to control the entire botnet infrastructure.

2. DC Darknet

The DC Darknet is a group of challenges based on the books Daemon and Freedom written by Daniel Suarez. At DefCon, agents of the Darknet fight to gain reputation points as they learn new topics and explore quests ranging from breaking ciphers to building simple exploits. The Darknet badge was one component of these quests.

This badge had a do-it-yourself element. The Darknet badges taught me how to solder, and now I bring a soldering kit to DefCon just to rapidly assemble the Darknet badge. There are a hundred stations in the Hardware Hacking Village but lines quickly form and who has time to wait for a soldering station? A quick 40 minutes after receiving mine it was assembled, flashed, and ready to start speaking with other agents.

A particularly interesting feature on the badges is the IR and RF pairing. After you built your badge, it could be pairing with IR to other agents which would allow for you to send RF messages to them wirelessly. You could state “I would like a taco, ” and that message would be relayed over to the agent of your choice(if they were within range). This feature adds a unique covert method to communicate with your new friends and fits in with the story extremely well.

The dialer aspect of the badge was a refreshing throwback. However, it was somewhat difficult in practice. I felt during one quest requiring a few key numbers (Emergency, Jenny) the touch capacitors would sometimes read incorrectly. Not having a backspace button can be incredibly frustrating when digits sometimes worked and didn’t work.

The team beyond the badge was equally as impressive. The Darknet staff table easily had ten staffers there at all times helping agents trying to complete quests, re-solder badges, or get points from the scavenger hunts. Another particularly nice touch was the rechargeable battery that helped me cut down on AA batteries and the need to charge them.

Although I did not have as much time to devote to the quests, I was able to participate in the boss fight. Working together with a group of people in a hotel room to go through quests was certainly one of the high points of this year’s experience.

3. Mr. Robot

DC Darknet and AND!XOR had both presented badges at DefCon, but the Mr. Robot badge was a cryptic newcomer. There were no official Kickstarter or starting quests to get the badge. Instead, you had to follow a minimalistic twitter page to find out where to purchase the badges and what they even did.

It was pretty amusing how they were handed out. The first batch was distributed out at skeeball which had a feeling that was similar to the show. However, I found out about the drop 4 hours later. I was luckily able to get a badge because I saw a tweet about a sale nearby Caesar’s when coming back from a party. The tweet only stated they were at the Spanish Steps and I stepped it out to get there as fast as I could without running. They were easy to spot because a woman with a large purse was looking around nervously while sitting with three other people. Nobody else had bags large enough to carry the badge. So in what only could have seemed like a drug deal, I approached her, slipped her cash and received my badge.

This badge has a beautiful mask and looks amazing. On the outside, it does not look to be as flashy with LEDs and only had two games (snake and Tetris) on it. Even then the up arrow froze the game. While there was additionally tweets for an ARG, I did not play with them much. Therefore, I was shocked when I suddenly saw a group of open wifi signals while connecting to the network. Later I went back and logged onto these signals to discover a wifi network with being the only host. When I unplugged the batteries, the wifi signal disappeared, and suddenly I understood it was coming from the badge!

So I did what anyone at DefCon would do. I logged back in and scanned the network for more devices and open ports. It bizarrely only had one open port UDP 4096 that was open. Despite trying to netcat and run commands against the port, I got nowhere. More discouraging was whenever I saw someone with the badge they knew nothing about the port or how they were carrying around a wireless access point.

Warning FUD and conjecture ahead! There are some rumors that the Mr. Robot badge also had a botnet component to it that would use this port. Once one received the code, it would look for other badges to trigger their code and then launch deauth attacks against other wireless devices in the area. The badge wearers, unaware of they were transmitting wirelessly, would walk around deauthing devices and could be spreading the virus across the conference. Right or not, it sounds like a fascinatingly devious scheme.

But these are just toys?! What does this have to do with security?

The great influx of badges added an interesting IOT component to DefCon. It is easy to forget that these badge designers were able to do amazing things on a tight timeline with relatively cheap devices. As businesses are exploring how they can do more things with the IOT, we will see more and more professionals coming up with outlandish ideas to do many more elaborate things. These are quickly built use cases of how the IOT is both incredibly easy to implement and how the best of intentions could create a raging multi headed botnet if you are not careful.

It was incredible to see the different layers of people coordinating across the country to pull this off, and I am very excited to see what they will put out next year. Who knows, maybe next year I can get a Texas badge put together!

If you want more articles on badges I suggest this one and if you are looking for an audio book I suggest checking my book on Effective Threat Intelligence.



My four leadership hacks as Harvard Business School get personal

As many of my classmates have already pointed out (here, here, here and here), our experiences at Harvard Business School during the second half of Professional Leadership Development program focused less on technical knowledge and more on the understanding of our personal attributes.

1. Lockpicking: Are you teaching criminals?

A major change in my attitude was to get out there and teach my fellow executives. What is something simple that I could uniquely offer? Lockpicking. I taught lockpicking with TOOOL for 3 years at DEFCON and have enough training tools that can fit into one of Emily’s old Clinique bags (people are less likely to walk off with an orange makeup bag). Everyone picked at least one lock with Nofi being a particular all-star that opened all the locks I had to offer.

Overall, lockpicking was a huge hit. I was able to host 3 different sessions and 40+ classmates learned something brand new. However, when posting on social media questions about teaching “criminality” arose. This seems bizarre since our lectures involve several in-depth discussions about fraud orders of magnitude above what anyone would ever see from home burglary. I think the critics miss how the takeaway for the students dovetails nicely with our studies. Locks provide a simple tangible process, which is much less abstract than financial fraud, which can be subverted to do something that wasn’t originally intended.

Another benefit was that after our short sessions and I was suddenly inundated by more complex security discussions. Topics were varied including advice for which security vendors to consider, the importance of password management, basic cyber hygiene, and even in one case advice on how to fight a phishing campaign. Without deciding to be a teacher on small things I wouldn’t have been able to drive a conversation on the tougher stuff.

2. Acting with the Ariel group

I never thought I would be taking an acting class. I was in small plays in High School but I never really felt it as a calling. However, on Saturday we all were sitting around being put on the spot for displaying emotions in improvisational scenarios and yelling “hah” at each other. I even told a story about my make-believe cat grooming salon. Most of it reminded me more when I sang in the Navy but we also learned an important framework for telling stories.

I don’t like telling stories and having something both relevant and impactful is tough. A framework was provided to keep things short, maybe ~2 minutes, and telling something visually interesting was a very good exercise. However, I didn’t really buy into it until I heard Noah from my live in group tell his childhood story. It was amazing and left us with goose bumps. As a Quaker, he spoke of fire, fear, and rebirth making me want to jump up and do something, anything, to help him out. His very personal story convinced me of the power of storytelling. Now I am looking at compiling a short set of stories to keep for leadership challenges.

3. Running a case study at AIG: Tunneling my inner Tushman

A very powerful thing we do at AIG is teach what we have learned from our professional development training. There are two benefits, obviously, our team can benefit from the information of an event. More importantly, the attendee is able to summarize what they have learned which galvanizes and better retains that information. So for my training, I purchased a few HBS cases, ordered pizza, and sat 20 people down in a room to go over a case.

It went amazingly well! Our diverse group argued and had healthy debates about the situation as I moderated frantically trying to keep up. I remembered the way the HBS professors would give equal time to both sides, raise pointed questions, and stop people from dominating the conversation. I didn’t even need to cold call anyone! There was always an opinion out there. By mimicking the behaviors of the professors I might not have been able to give a true experience but it got lots of people interested in taking HBS classes.

4. All the sports you could muster

At 3 am I woke up to watch a Rugby match of the AIG All-Blacks playing the Lions. I’m not really good at team sports. When I played soccer I would play with the grass and stare at the planes landing more than the ball. So waking up at 3 am to watch what my roommate Paul said was “a huge match that only happens once every 7 years” I wasn’t that excited but I knew he was I sent a message out to the 140 cohorts inviting us to join, set my alarm and went to bed.

When I woke up at 3 am the lights in the living space didn’t automatically turn on. Even the building knew that it was too early for anyone sane to be still up. Nobody else showed up, but Paul and I were there sitting in the dark, eating potato chips, and going over the finer points of rugby. While I was exhausted the next day, it was extremely fun to see how the match played out and see the AIG All-Blacks pull a huge win.

There was also a baseball game where we watched the Boston Red Socks and suddenly our roles were reversed. I was the sporting expeert and knew tons more about the game than many of our international cohorts. I spent time discussing how strikes, outs, and innings worked to people.

So why were sports important to my studies? I feel it goes back to how important it is to be willing to step into the roles of both a teacher and be a student in groups. Learning something from someone, even something very simple builds comradery, and trust. When a more complex topic comes up we have the tools and relationship to handle new challenges. Sports offered an easily useable stepping stone to deeper conversations.

So here about a month after I left Harvard I have been thinking more about my roles as a teacher, actor, and sports aficionado. Its been very bizarre and almost a completely different experience than the first segment at HBS.I find myself very appreciative to Harvard Business School for designing a program I never knew I needed and very curious on how the next module will transform me.


Conquer creativity; seize your three golden hours

Budgeting time is a universally popular topic amongst management courses, linkedin articles, and blogs. Time is the universal resource people trade for fun, salary, and sleep. Companies routinely judge employees by the amount of time completed on the job. Man-hours are formally tracked for many companies, work-hours are often explicitly stated, and even informally we are aware of our colleagues who show up late or waste time in meetings. An entire article can be written about how time management is a key component in professional training courses ranging from the military to medical school. Everyone is looking for a way to hack out some additional time in their day.

But I suggest taking a closer look. There are two separate time management issues at hand. The first is the classic triage time management approach of treating work as having different levels of importance. A kid being rushed into the ER is more important than finishing a level on a video game. However, in my own journey to perfect time hacking has led me to a second realization. Certain work is best matched with certain time and understanding that will make you the most effective for yourself, your boss, and your loved ones.

As I was watching my daughter try to shove little triangles and circles into matching holes I realized one of my biggest problems is a misfit of tasks and time. I am less effective when trying to accomplish tasks without properly matching up the best time. Throughout the day my hours have different constraints, I have different emotions, and distractions that impact effectiveness. Intuitively this makes sense. Is 2am a bad time to host a meeting? My co-workers will be grumpy. Should you plan that meeting during the family dinner? The wife starring you down will be grumpy. Are you going to get your best work done at those times? Probably not. But these scenarios are easier to understand because we typically understand if we are inconveniencing others and it is not the best time for them. It is much more difficult to be introspective and discover at what time you will be the most effective.

Different time means different things

I group my time into 3 categories. Creative, routine, and rest. Each one has its own little attributes and quirks that best fit certain tasks.

Creative time, these golden hours, are the most limited and productive times of the day taking swaths of brain power and concentration. This is the good stuff that builds to Gladwell’s 10,000 hours of mastery… and it is draining. Its enjoyable because when I’m there it is almost a trancelike state where I am only concerned about what I am doing in the moment. If everything goes correctly I will slowly shake out of my trance relaxed, tired, and with a sense of accomplishment. Alas there is always some micro disaster going on wanting to pull me out of this state. While there are some ways to eek out more of this time by eliminating distractions (hooray headphones) it is always limited. On a good day I can get about 3 golden hours and my average day will knock me down to about an hour.

Routine time can be seen as a very watered down golden hour. This is basically the 5-6 hours a day where I am participating in meetings, drafting reports, and actively listening to problems. It’s not the best work, it’s not the most creative, but it gets things past the finish line or primes a project to launching point. The conditions are not as sensitive and I do not need to worry too much about being distracted.

Resting time doesn’t mean my brain isn’t working. It can just be extremely passive. Mindless television, driving, listening to audiobooks, big meetings, time with family, childcare, naps and knocking off items on the to-do list are all things that help clear my mind. Often we are able to multitask doing these things. Think of all the people listening to audiobooks on their drive. They do not have to argue with the audiobook or be concerned about optimizing their travel time on the road, they can just enjoy. I use rest time as a method to triage other tasks that require more attention at some point in the future. Audiobooks get bookmarks so that I can go back and listen to the information using my routine or creative time. One of my favorite leaders once said that he always went for a run with a problem in his head. He believed that the oxygen deprivation helped him come to solutions for when he got back to his desk. Thats the perfect use of rest time and I highly recommend it.

Fitting the right peg in the hole

The impact comes from how you can manage these three time types to get the most out of your day. Matching up your current work priorities (classic triage) and how you work best is an extremely difficult personal problem requiring good self awareness. So while results may vary, here is how I look at it.

My most important time is now that creative time that comes either at the beginning of the day or right at the end. My daughter is safe at daycare, no emergencies, and no meetings demanding my immediate attention. It is the perfect time to do difficult problems, get some serious work done, and give myself time to sink into that trance. It is rare, elusive and about impossible to get back into that frame of reference after I get ejected from it.

Routine time is the default time used at work. Its inconvenient to stop or start tasks, but not completely detrimental. I can easily handle small emergencies or shift to tackle small roadblocks that have popped up for the team. Small tasks, hiccups, and work roadblocks that you can do without much thought or concern on autopilot can stay under routine work. The real value of checklists and processes are that they turn the unique into the mundane repeatable items you have done millions of times. When they become so ingrained that no new decision needs to be made it can be completed in your rest time.

Rest time makes up the remainder of the day. There is no need for ritual or preparation and it is best for small tasks that can be broken down. When people are ready to “veg” they are doing these things and are tasks that you can be running on autopilot. Reading light literature (blog posts), sorting emails, swiping left and right on mobile apps, paying pesky bills, doing chores, catching up on your favorite TV shows or with family and friends all fall into this category. If you are tired, unenthusiastic or in bad spirits these are the activities people default to. The most important thing to realize is thats ok. You can’t run full throttle all the time but it is good to place complete your simple tasks. By getting things done you can stay motivated without losing momentum or your mind.

Building a nest for those precious golden hours

DO NOT GIVE UP YOUR GOLDEN HOURS! Fight for them with every action of your day. It will give you that creativity, the work boost, and that sense of accomplishment that will carry over the rest of the day. Set up conditions so that your golden hours will be the best uninterrupted work it can be. Also don’t give it all just to your work. Spend some time working on some personal goals and investing in yourself.

I need some prep work to build my nest and get into the zone. I need to have some coffee, find a new area away from distractions, and just go into tunnel vision. My golden hours are so important that I use my routine/rest time to prep for them. I setup playlists, clean up the coffeemaker, and move things into position so I will not have to be interrupted. I schedule meetings and anticipate problems to make sure updates won’t occur during my golden hours. But that is just me. I know other people who need to have just come out of an exciting meeting or workout session to ride that post accomplishment bliss sitting in a rat’s nest of a workspace. Whatever works. The important part is that you are realizing which time space you are in and which work is accomplished best there.

If you want to identify your golden hours I suggest this:

  1. Describe what your golden hours look like? When do you get your most effective work done and what does the environment look like?
  2. For one week see if your hypothesis meets reality.
  3. For the next week, challenge yourself to add 15 more minutes of that effective work time and keep a quick log of what you get done.

I have found being able to tap into my golden hours has helped me achieve much more than I thought possible in a day. While most of our days aren’t optimal, we should seek to integrate time management to match our most important work to the most effective time slots. We should strive to learn when our golden hours are and how to build ourselves nests to capitalize on them to increase creativity. There is no time to waste to do great and interesting things.

Additional Reading:

If you enjoy these articles please let me know by liking and sharing them below!


Five reasons Harvard Business School executive education beats out traditional IT certifications

Last September I did something I thought I would never do.

I applied to Harvard Business School.

I never really thought I would be attending. My grades/rankings through high school and college have always been good, just not Harvard good. Additionally, I never pursued the typical route of a Harvard student. I have read more binaries than cash-flow statements and choose discussions on botnet takedowns over hostile takeovers. So being accepted to Harvard Business School’s 6 month PLD program surprised me greatly. Having just completed the second module of Professional Leadership Development I can think of several reasons I am glad I chose this course over a typical IT certification.

1. I did not understand Business

I thought I understood. All my certifications and experiences had me rabidly supporting the notion that the business is king. However, this sentiment in certs and work were just small two sentence blips shoehorned into the much larger core security concepts. Overtime, I grew a bias that my supporting work was essential and everything else was trivial. You could always find someone to do marketing or sales, but IT professionals are enjoying a negative unemployment rate and are swiped up fast. I was much more likely to crack open a book on encryption instead of on business because of my mistaken belief that “business” was easy.

This shifted as I looked for ways to use my Post-9/11 GI Bill. Structured courses have always been a great way to prime the learning pump for topics I didn’t completely understand. As I looked across the diverse range of MBAs and EMBAs, I came across the Harvard Business School’s Professional Leadership Development program. It was a fantastic alternative to other Executive MBA programs for me.

With no understanding of finance, marketing, or accounting, the HBS online curriculum provided immediate immersion for me on day one. I was floored by the presentation and the complex concepts conveyed in a simple manner. The topics got much more difficult over time and I’m not proud of some of my scores, but I learned. I even started to enjoy it! Now a financial statement book is sitting on-top of my reading list.

More importantly, I have learned that businesses are fascinating in ways I had never considered. After staring at cash-flows or income statements for anomalies I started noticing little weird discrepancies popping out at me. The aha moment of seeing aggressive accounting is a very similar feeling to the joy I experience when I find a hidden instructions in malware.

2.    Real understanding of actual business problems

Harvard is famous for their Case Studies that are a fundamental part of the course. Before you arrive on campus for module 2, students read around 40 cases on various subjects. As the HBS website describes it

…the case method is a profound educational innovation that presents the greatest challenges confronting leading companies, nonprofits, and government organizations—complete with the constraints and incomplete information found in real business issues—and places the student in the role of the decision maker. There are no simple solutions…

Cases are tough and I found them extremely fun (in some sick twisted way). Personally, I covered each case three times. I read them the first time to understand the basic concepts, I discussed them the 2nd time with the live-in group, and then finally with a professor in the classroom with 80 of my colleagues each giving unique insights. Every time I learned something new and had a completely different perspective on the case.

The course throws you into the core of problems of businesses we know conversationally. Lots of us have heard of Enron’s downfall, devour Apple products, wear Lululemon, and some may have shopped at Cardullo’s. Each of these cases are relatable enough that you have probably discussed them with friends but the case studies take it to the next level. Each one puts you into the shoes of an executive at a critical decision point. I won’t call it LARPing for business nerds… but it totally is. The case studies we explored provided a different perspective of how an executive led a company to make an extremely pivotal decision. Understanding the issues, deciding how you would act, and seeing the direction the company took is excellent preparation for the next big decision you will make.

3.    You connect with a group not your own merits

A big reason I chose this course was the living group arrangement. Back at the Naval Academy we all lived in one building, Bancroft Hall, and these friendships continue to . HBS does something similar but instead of Bancroft we have Tata. Our groups might be a bit smaller (7-8 instead of 40) but I feel connected to them in a similar manner.

The diversity of these groups is amazing. In the program of 160 only 50 are American and that flowed down into our small groups. My group alone (the prolific 5C) represented Canada, Czech Republic, Kuwait, South Africa, Turkey, and the United States.

In two short weeks:

  • We studied and suffered an incredible packed curriculum for hours.
  • We argued over excessively over cases. Perhaps investing a little much in events that have long since passed.
  • We had our own inside jokes and terminology about chicken banks, “Lars”, plumbers, and other key players from cases.
  • Some of us were cold called (called without warning in class) with very tough questions
  • We even discussed the challenges of being a working parent

I believe The added stress in the environment pulled us together as we didn’t want to let each other down. Plus each of us had our own unique experiences and expertise to add to the group. I would have never understood important aspects of the cases if it weren’t for my fellow members and our short timeframes.

It was a fantastic experience and essential part of the program that is hard to replicate.

4.    Meeting the Former Secretary of the Navy

I never expected that I would be able to meet former Secretary of the Navy Ray Mabus. But in-between classes I was asked to step out with other Navy veterans to speak with him. It was strange to sit across from a man who drastically shaped a majority of my Naval career. More impressively they even invited past Navy PLD Alumni to the small discussion.

The program didn’t have to go out of their way to coordinate this event. Nobody would have noticed if this didn’t occur. It is this unexpected perk and service from the HBS faculty that really drives home their commitment to their student’s current and future development.

5.    Anticipation for the future

The best part of the entire experience is that there is so much more to come. We are only half done. The next module has lots of work for us to accomplish including a personal case, an alumni case, individual development goals, and of course more case studies. I really miss the dedication of the people I was with these last two weeks and am looking forward to seeing them all again in just a few months.

I could not be more pleased with my decision to pursue my goals with Harvard Business School. Now I have a whole host of new tools to push my development in surprising new ways. With this new found experience I feel that I will be equipped with better skills than any IT certification could provide.

One parting word of advice…

Beware the chicken banks


2016 Hacker Santa and the joy of sharing

Coming to the close of 2016 @InfoSystir was kind enough to set up #HackerSanta which provided an infosec goodie exchange.

My target was @J0hnnyXm4s a hacker who has done numerous infosec talks and helped discuss the risks of TSA keys. After some stalki… open source investigating, I determined that he was probably in the market for a new pelican case, enjoyed lockpicking, and would likely appreciate something repurposed.

Luckily I had a pelican case, some locks, and some things that needed to be repurposed. I chose two locks to secure the goods. This Masterlock is a favorite I have kept around to help humble lockpickers when it became “too easy.” A spare TSA 007 lock would serve as a tip of the hat to Johnny’s research. Everything was coming together nicely.

DomainTools promotional video repurposed

Back in February, DomainTools was kind enough to send me a promotional video in a unique package. The video started to play as soon as you opened the cover. While they have a great product (I encourage everyone to check them out,) I only needed to see it run so many times. So how does this work and what can I use it for?

At first look I could tell there was some type of switch to set it off, a small usb, the controls, and a hidden speaker. However, there was not much else to go on. Time to take it apart! 10 minutes of cutting through adhesive later it looked like this.

Pulling information off the board I was able to find a company selling similar products if I needed to use reference material. However, this was not needed since the USB provided a simple interface and not just for power. After plugging it into my computer I discovered the advertisement was a stored mp4. It was just a simple swap to put in my own mp4.

Fast forward to December, I felt providing a video to Johnny for getting through the locks would be great touch.

The Game is a foot!

All packaged together (including shameless self-promotion) it arrived a little after Christmas. Although I sweated over battery life fears, I was very pleased to see that not just Johnny but a whole host of @BurbsecEast attendees also had fun getting the locks open! Its great to see people come together to solve my silly gift.

Finally it is opened!

So besides from a long awaited Rick Roll you can see how I put it together. The switch is attached to the lid but the magnet is not strong enough to lift the whole screen. Everything sits on top of the foam which will allow Johnny to repurpose both the video player and the pelican case.

What’s the deal with the coin? Ask @Curious_Codes. I received it after completing one of their puzzles at Derbycon and wanted to share the joy I got from it.

I had lots of fun putting this all together. Thanks to @J0hnnyXm4s for providing great documentation of his process and sharing it on twitter. @DomainTools for the promotional hardware. @InfoSystir for setting HackerSanta up. @Curious_Codes for the puzzle within the my puzzle. My HackerSanta @Greenjam94 for the TV-B-Gone. I am excited about next year!