You've got mail!
Mail delivered yet?
I've been using the 'Informed Delivery' service from the US postal service for some time. It's a nice service where they email a picture of most of your mail every day. Recently, I was eagerly anticipating a check in the mail and spotted it in the day's informed delivery email. I trekked out to the mailbox a couple times looking for it. Finally it came but it got me thinking: it would be nice if the home automation dashboard showed mail arrival.
First attempt (that didn't work)
The easiest thing I could think of was to put a motion sensor in or around the box to alert on nearby motion. But if placed inside the box, no signal was sent and if placed underneath the box, there were too many false positives. I couldn't determine whether the motion was from the mail carrier.
Idea!
I knew that it was possible to detect a particular object in an image or video and I already had a security camera pointing out to the driveway (and also the mailbox) so if I could detect a mail truck out at the mailbox between certain hours, I could assume that the mail was delivered.
Step 1 - Pull images from the video feed
From using FFMpeg in the past, I knew it was possible to pull periodic images from a video feed. Specifically, I knew how to pull images from an RTSP feed. The problem is the camera that I have pointing at the driveway was a very reasonably priced yet feature-packed Wyzecam. But the Wyzecam was using proprietary software and I wanted a way to access the feed to extract the images. A while back I had seen a project where someone had created some 3rd party firmware to replace the default Wyzecam firmware so that it would become an RTSP camera. So I searched for that project. But instead I landed on a page created by the Wyzecam company itself providing a firmware replacement that would make the camera RTSP enabled. And it's fantastic. Not only does it still work in the existing Wyze cam application but it gives you an RTSP url access to the feed. Just what I needed!
My first attempt at image extraction was to use FFMpeg but for whatever reason it just wasn't reliable. It kept hanging up on the extraction without an error. It would just stall. So after more searching, I landed on a solution where I could make a command line call to VLC and pull a snapshot every x seconds. From watching the mail delivery, I knew that a 5 second window was enough to get 1 or 2 shots of the mail delivery. He's fast but 5 seconds is enough lag to catch him.
Once I got the VLC snapshot call working well, I wanted to schedule it. I knew the mail usually came between 10 am and 1 pm Mon-Sat so I setup a launchctl program to run on those hours. I'm using an m1 mac mini for this but if I was using Linux, I would use systemd for the scheduling.
Step 2 - Use artificial intelligence to identify the mailtruck
I had previous experience using Darknet by PJReddie to identify humans in the motion detection of my existing cameras. But this time, I wanted to identify mailtrucks. I tried to find anyone else who had detected mailtrucks so I could use their files. But that search ended fruitlessly so I went about learning how to train the AI to detect a custom object. I found this Youtube video where the creator teaches how to detect koala bears. I followed his video exactly but inserted mailtrucks everywhere he configured koalas.
The hardest part of the training is installing the labelImg software to draw boxes around the custom object in question for 300 images. At first, I drew boxes around all the mailtrucks in 300 images. But when I ran the detection among a group of photos, it was identifying every box truck as a mailtruck. The solution ended up doing the same process but this time drawing a box around the mailtruck postal logo - not the entire truck. So again I went through all 300 images drawing boxes around the mailtruck logos. And now the detection worked!
Step 3 - Show the mail delivery on the home automation dashboard
Whenever a mailtruck is detected by the system, the image and the time of delivery gets sent to an MQTT server with a particular topic. The dashboard application, which is a Phoenix application running on Elixir, subscribes to that notification and when it detects the picture and the time, updates a block on the dashboard that shows the actual image and the time of delivery. Every night at midnight, this data is cleared so that the block shows a waiting for delivery message. The next delivery day, the process starts again. I thought about just having an icon to show the mail delivery but I wanted to know for sure the system is working and the way to know for sure is to see the actual photo that was taken revealing the mail delivery.
Tada!
