Summary
- Most false event cards were removed by the zones the head of HACCP drew. We only touched the model thresholds after that.
- The plant decides what counts as a violation. For us a crate on a pallet, trolley, pallet jack, or stand was normal, a crate directly on the floor was a violation, and if the support could not be seen, the system marked it "undetermined".
- Accuracy on a small sample misleads: on 45 frame crops we got 47% wrong verdicts, while a blind check of all 219 crops (131 events) gave 23%.
- A rejected event card does not yet mean there was no violation. Often it is just a duplicate of a scene that was already confirmed.
- A new rule runs in shadow mode first. A rule that has already proven itself in live mode goes straight to live on new cameras.
For: heads of HACCP, process engineers, and owners of meat processing and food plants
Where we started
The first plant where we ran DuoVision was a meat processor. We did not install new cameras there: we worked with the ones already hanging in the deboning areas and on the loading docks. The system took video from them and checked the rules one by one: hair cap, face mask, gloves, crate on the floor, riding a pallet jack.
Each detection became an event card with a frame. The head of HACCP opened the card and pressed "violation" or "reject". His decisions were our starting point when we calculated how useful a rule is. There was no automation at the plant that punished people: the camera found the situation, and a person decided what to do about it.
Below is what we did not know at the start and now do at every new site.
Zones do more than model tuning
In the first weeks we tried to remove false event cards the "smart" way: we adjusted thresholds and added filters based on a person's size and distance from the edge of the frame. The head of HACCP looked at this and said roughly the following: zones are drawn on the frame where the model should look, and a person without a hair cap standing far away will be seen by another camera.
He was right for two reasons. First, zones are drawn by someone who knows the production floor: where the tables are, where the walkway is, where a person is visible only from the waist up. That knowledge is more accurate than any formula about height in pixels. Second, cameras on the floor overlap, so if you drop a bad spot from one camera, the person still ends up in the frame of the next one.
Now, at a new site, we first draw the zones on every camera together with your responsible person, and only then look at the thresholds.
The plant decides what a violation is
The model sees objects. Whether there is a violation depends on the rules of the specific plant, and these rules have to be discussed before launch. Here are the cases we discussed for longer than we spent configuring the code.
| Situation | Decision | Why |
|---|---|---|
| A crate of product on a pallet, trolley, pallet jack, or stand | Normal | The product does not touch the floor |
| A crate standing directly on the floor | Violation | Risk of product contamination |
| The support under the crate is hidden by a person or another crate | "Undetermined" | The system does not guess what is underneath |
| A person standing on an electric pallet jack with a platform | Violation | The head of HACCP decided so after reviewing frames |
| A forklift with the driver in the cab | Normal | This is standard equipment |
The electric pallet jack case shows well why this should be discussed in advance. We assumed that standing on the platform was allowed, but at that plant it was forbidden. If we had decided for the plant, the rule would have stayed silent exactly where it was needed.
Why accuracy "by eye" cannot be taken as the truth
The first question about any rule is: "What is the accuracy?" We also wanted to give one number. A first estimate on 45 frame crops suggested that almost half of the "glove present" verdicts, 47%, were not supported by the frame, and we nearly started reworking the rule because of it. Then we blindly checked all 219 crops, 131 independent events: such verdicts turned out to be 23%. Same rule, twice the error rate: 45 crops were too few to draw a conclusion. When we reviewed 8 frames ourselves, all the disagreements were on the borderline between "glove is there" and "cannot tell".
The second trap became clear when we asked how the head of HACCP judges event cards. He pressed "violation" if there was a violation anywhere in the frame, even at the next table outside the box the system drew. For the plant this is correct, because the violation is recorded. But it means that the share of confirmed cards shows only the upper bound of a rule's accuracy. The real hit rate of the box is lower.
What we do now:
- we always give a number together with the number of frames it was measured on;
- we compare rules with each other on one and the same set of frames;
- for a number that goes into a report, we use log records, not impressions from viewing.
If a vendor quotes you an accuracy figure without a sample size and without saying who counted it, ask for both.
Rejected does not yet mean there was no violation
A violation rarely lasts one frame. A stack of crates on the floor can stand there for an hour, and the rule will create several identical event cards. The head of HACCP confirms one and rejects the rest as duplicates.
If you count every rejected card as a system error, the rule looks wrong more often than it really is. So before any measurement we group event cards into events: same camera, same spot, within about an hour. If an event has at least one confirmed card, we do not count the remaining duplicates as errors.
The event card queue and a filter in front of the panel
When the number of rules approached thirty, the queue of event cards for the head of HACCP started growing fast. Some cards were obvious junk: for example, the model took black holes in a table for knives.
We put a filter between the rules and the panel. It looks at each event card and sorts it into one of three groups: "violation", "no violation", "hard to tell". The filter deletes nothing, it only sorts. Doubtful and junk cards stay in the log and can be opened, and the person first sees the ones where a violation is most likely.
To us this seems an honest compromise: the person responsible spends time on what matters, and if the filter makes a mistake, nothing is lost.
Shadow mode, and when it is not needed
A new rule first runs in shadow mode: event cards are written to the log, but nobody is disturbed. During this time we review true and false detections together with the person responsible and decide whether to switch the rule to live mode.
Once we wanted to run a rule that was already live on several new cameras in shadow mode in the same way. We were told, fairly, that there was no point, since it had already been tested. Now we use shadow mode for new logic. A tested rule goes straight to live mode on a new camera.
If you plan to do the same at your plant
A short order of steps we arrived at:
- Look at which cameras you already have and what they see. For hair caps and gloves, the person must appear large enough in the frame.
- Write down three to five violations that hurt the most, and agree together what counts as normal, what is a violation, and what is "undetermined" for each of them.
- Draw the zones on every camera. Your process engineer or head of HACCP will do this better than any integrator.
- Run the rules in shadow mode for a week or two, review the event cards, and only then switch to live mode.
- Agree who judges the event cards and how accuracy is counted before the first report.
If you want to see how this would look at your plant, we can connect to your existing cameras in shadow mode, or run footage from one camera through the system and show you the event cards it produces.