This short story was written by Danni Youziel as part of our ODI Writers' Fund for Black History Month 2020.
About Danni
Danni is a software developer with interests in data and security. She has spoken at Black Hat Europe about bias in biometric authentication and impact on people and their privacy. Danni has been fascinated with this area of technology ever since she saw someone pay for coffee with their face. She is passionate about diversity in tech and is a mentor as part of Coding Black Females.
Facing Decisions
There is no cure that does not have its price.
Kenyan Proverb
“Mama Judaae, the journalist just called, he will be here soon.”
Startled, she turned as she was brought back into the room. Calming and quiet, her study was an easy place to lose yourself. A plate of deep fried plantain lay untouched beside her chair. The scent of nutmeg and cinnamon no longer filled the air.
“Assante,” she replied as the food was taken away.
To look at her, Judaae was a picture of wisdom and elegance. Her black braids, now streaked with grey, were piled on her head, with a sea blue wrap holding them in place. Her wrists were bare but for a few rose gold bracelets to compliment her caramel skin. Today, however, the focus was her legacy, her company. Devision, ‘Turning Data Into Decisions’. The company’s newest technology reduced hundreds of millions of facial expressions to a set of emotional and behavioural traits. It could be used to detect scared children in a crowd or whether a suspect had the potential to become volatile. It was built to help people make fairer decisions based on data, not just instinct. Now, as the technology was officially a success, it was time to tell the Kingdom of Equatoria what she had achieved. And she made sure this technology enthusiast would tell the story through her eyes.
“Salaam,” she said, directing Saamir to the chair opposite hers.
As he sat, streaks of sunlight illuminated his fair-toned cheekbones and made the edges of his thick black hair appear golden. Saamir liked the view – its brightness was inspiring. The Council’s drones, jet black, whizzed past, keeping an eye on the city. Neat rows of commuters filled the hover lanes and quickly recovered from any chaos caused by the less experienced riders.
“Apologies Mama Judaae, there are multiple diversions due to the memorial. My GPS did not update.” He spoke with a gentle smile.
“A memorial. Such a waste. It was years ago. The citizens really must move on.”
Unsure of the appropriate response, Saamir cleared his throat before confirming receipt of their collaboration agreement. Before today, Saamir outright refused to allow subjects of an interview to sign off on his final pieces. However, for this he made an exception.
“Quais Mama, where shall we begin?”
Where do things like this start, he wondered. Can you actually pick a point in time? Most Equatorians could not remember life without their clones. It looked much like what the elders sometimes referred to as a ‘smart watch’. A watch, he laughed – he had seen one of those once. His clone was so much more. An oval onyx disc could tell him what he should eat, what film to watch, anything. The more he used it, the more it learned, and it was secured by the latest in biometrics. Every normal Equitorian would queue for hours for the newest version. Everyone in his circle except Kwaamae...
He recalled a conversation they’d had: “Wahlawi Saamir! You are just giving away everything about you... and for free!”
“And what if I am?! It’s my data and I have nothing to hide… do you?” On that day, he did not have the energy to defend his world, but regretted his insensitivity. Both felt an unfamiliar tension between them after that fight.
Judaae glanced sideways, as she thought back to an evening with a dear friend. Asaame, Head of the Equitorian Guard, had baked a batch of golden tahini puff puffs. He came to visit because he desperately needed her opinion. After too much sugar and honey wine, the pair ended up discussing the recent unrest.
Asaame’s job was becoming harder. The drones could identify the rebels that the Guard already knew about. Officers were then sent in to neutralise those individuals and keep protests calm. As tension began to rise, there were more and more protests. Some rebels could not be identified by the drones. His officers were going in blind, into situations where they had to make split-second decisions. They were overstretched, tired and acting on instinct. They were good people. Asaame was outraged each time he had to defend an officer’s decisions.
Judaae was unable to hide her surprise. As an elder of the Junabian, he imbued justice in his golden printed robes. A very picture of strength with a mahogany-toned bold forehead. As her friend, he was lost.
She directed her attention back to Saamir. “Chief Asaame did not know what to do… that is when I decided. Facial recognition was not enough, the government drones needed behavioural profiling too. Everyone does it, I decided to automate it.”
This is the moment Saamir had been waiting for, the question he’d crafted since the day he received her invitation. “How could the drones profile behaviour when their facial recognition was flawed?”
Judaae leant forward, and replied with ease. “By correcting Xcom’s mistakes.”
There had been much controversy over the use of facial recognition. Some states in Equitoria had banned it, but in the city it was still in use. The Council had chosen Xcom software, it was developed by a company based in Maisaira State.
Kwaamae was not a fan. You could always tell. Whenever he spoke about Xcom’s latest update or tester events, her eyes narrowed to nothing and ebony deep skin became more intense.
“I don’t need one of those things calling me kawaager too! Dark skin. Not feminine enough, not masculine. That makes me intimidating, does it? That makes me scary?!” She raged.
According to the press releases, Xcom’s revolutionary technology was based on an algorithm trained to look for features that define a face. Eyes, the edges of a mouth, and of course a nose, classifications. It then put classifications together and determined if the face was male or female. However, as Judaae explained, Xcom overestimated the diversity of the training-set data. The matches weren't 100% but a probabilistic match based on the data used. In some cases someone's facial features could not be classified as male or female.
“Xcom’s technology could mislabel people or not label them at all,” she concluded. “Some people’s features are different... too different. Those people have always been known as Kawaagens. Yahni, it was blown out of proportion!”
Saamir could not help but think of Kwaamae as he asked, “How often did this happen? How often were people falsely classified?”
“Moto yangu, adjust your view, society cannot accommodate everyone. There were many trials. The Council concluded Xcom’s technology was acceptable for use.” She exhaled slowly. “I believe maybe 45%. There was a 45% chance a Jinkan woman would be identified as a man.”
“As opposed to…” His tone, firm.
“As opposed to 1% for a Jorubian.”
It was starting to get dark outside. Without thinking Saamir scanned his clone, then paused. He knew Xcom’s classification was not always accurate, but not by that much. He had never been mis-identified.
“And… well, how did Devision fix the mistake?’
Judaae stopped to think for a moment, the moment that changed the game. She decided to synthesize human faces to increase the training-data sets. Back then she had a small, loyal team. On her instructions, they collected images of a standard Equitorian face. They then narrowed eyes so the image appeared like someone of the Masairan tribe, or widened the jaw to emulate a Jinkan. The team generated equal numbers of faces for each tribe so the dataset was diverse, no longer biased. Once Devision was able to build a recognition system that was 10 times more accurate than what was available, the emotional profiling began. The team began to classify faces in terms of emotions and train an algorithm to detect those emotions.
“They were able to detect four categories with confidence in their test data. Anger; agitation – the ones the Guard needed to calm the unrest. It was… it was an amazing achievement!” Judaae exclaimed. “From there we started to map emotions to likely behaviour. We were determined to calm the civil unrest.”
Saamir, sat silently, unsure. He had an image of people creating a random face Maisairian out of nowhere, making it angry, then asking a computer to confirm what they had dreamed up. It was all a bit much. He remembered when first hearing of this emotional profiler Devision built. All media spoke about identifying vulnerable people as well as threats to social cohesion. The work was extraordinary, but he was beginning to think it crossed ethical lines.
“How was that possible, mapping emotions to behaviour? If you don’t know what caused the emotion, how could you know what will happen next? Where is the science?” Questions just came one after the other, as he struggled to make sense of this.
Judaae tapped the edge of her chair, nails bouncing off wood. “Remember, I said mapping to likely behaviour. For instance, how many people, how many groups did you see? If you saw a group and the majority looked angry, don't you think you could predict what that group might do?”
She was not making a statement, she was stating a fact. Devision’s technology worked, The Council praised it, the citizens loved it and Equitoria was at peace. Hands clasped, temples strained, her energy was a little drained. This writer was failing to see the beauty, the peace she had brought their kingdom.
“Trial and error was part of the process. The most important thing in all of this was getting the right result.” She was relieved to see the lines around his eyes ease.
Judaae explained how the team were keen to move beyond synthesized data in the next steps. Carefully, she outlined how the system worked in isolation in the lab, but was not as robust in live environments. “Crowds and poor lighting reduced the accuracy of the results... The team wanted real images, authentic data to see if the system really worked.”
Satisfied Saamir had refocused, Judaae carried on.
Devision created a trial, it ran for three months with a diverse and large range of participants. They were put into groups based on ethnic background, lived together, ate together and worked together. Their group leader was their only contact with the outside world. The mentors would build baselines through activities meant to provoke specific emotions. Devision programmed drones with their profiling technology. The team were able to monitor their mini society from the other side of the city. Every face, emotion and behaviour was analysed and classified.
“Now do you see? We were committed to getting this right.”
She had his full attention now, this news was new. There was no mention of this trial in Devision’s published research, or the trial outcomes.
“Where did this take place?” he laughed. “It sounds like you were torturing inmates...”
Six sharp bells echoed from a grand clock in the corner of the room. It masks the sound of nails hitting wood. Both were oblivious to a gentle knock before the door gilded open.
“Mama Judaae”, came a whisper, as fresh air cleared the stale room. “Would you like more tea?”
“No, it is getting late, my guest will be leaving soon. Close the door behind you.”
Judaae removed her bracelets, an indication the meeting was over. “Saamir, you must excuse me, I have spoken too long and must rest. Send any drafts to my secretary. Let him know if there is anything else you need.”
“Mama, one more question. Your story will not be complete without this.” He was buying time, he needed to phrase this delicately. “You must have done many trials, before the technology was approved. You wanted to be sure your system was right.” He paused. “Tell me, which trial taught your team the most?”
“It was a collaboration with Asaame, at a festival... Koruba”, she softened. “It was an amazing opportunity, my team got access to a perfect testing ground. They introduced audio to the profiling system.”
Koruba was a yearly carnival held by the Jinkas to celebrate their Gods. A festival of light, dance and music that all of Equitoria enjoyed. Saamir and Kwaamae went every year. Every time the pair ended up in the south-west corner, with a plate of kitfu and too much rum. According to Kwaamae, that’s where the true Jinkans partied. A few years ago, there had been doubts the festival would happen. In the end it did.
“There will be a real riot if The Council cancels Koruba”, Kwaamae had joked.
Saamir didn’t joke about the festival anymore, he had stopped by the memorial that morning. His throat felt tight. He and Kwaamae had arrived late that year, just in time to see the barriers go up. Later, The Council announced that the drones had identified an agitated, potentially aggressive, group of Jinkan’s in the south-west corner of the festival. To maintain peace, they put that section of the festival in lockdown until the Guard could get to the location.
It was only now, he realised: the Guard had used Devision’s technology that night. Saamir remembered the scene when the barriers at the festival were finally lowered. There was a disused factory, it was due to be demolished not long after the festival. It was unsafe, but an interesting space to party in. By the time anyone realised there was a gas leak, too much rum had been drunk. They poured onto the streets in a frenzy, disorientated. They tried to get to where it was safe, but the lockdown held them firmly in place. By the time the Guard reached them, it was too late.
“Koruba was unfortunate, but the team learnt a lot from it. They built an incredibly sophisticated system. Look at Equitoria now, we have peace. Don’t you see? Our society is strong because data drives all of our decisions.”