China Citizen Watch, the official Chinese division of the Japanese watch giant Citizen, and Bulova Watch Company (a Citizen brand in the U.S.) have both been affected because China Citizen Watch or its hosting company left an unsecured RSYNC server online with more than 150TB of files.
Cursory skimming of the files, necessitated by Citizen Watch’s repeated failures to respond to security alerts I sent to them, indicated that the RSYNC contained copies of backups from various workstations and email systems for about 500 internal Citizen employees and staff as well as many from Bulova. Some files on the system also contained usernames, emails and plain text passwords all saved in .csv formatted files with no encryption to protect them and no password required to access them. The bulk of the data on the RSYNC appeared to consist of email inboxes and all related data, Sent, Trash, Inbox etc.
The leak also affected Vagary.cn, Bulova.com.cn and various other small brands owned and controlled by Citizen.
Attribution
Attribution was fairly easy. Cursory inspection of some files revealed that their host was upmcn.com and that Citizen was likely the owner of the data. The email inbox also went to an internal domain registered to Citizen, and the inbox configuration all had Citizen smtp details within them, as illustrated in the figure below:
Tick Tock: Why Didn’t Citizen Respond?
The data was discovered on November 22, 2019. Contact to China Citizen Watch was first made to their Chinese email address within 48 hours. Citizen did not respond to the emailed notification, but I could see that my email notification to them showed up in their backup.
After one week, there was still no response from China Citizen Watch, and the data were still exposed. On November 29, a second email notification was sent to them. In addition, I attempted to contact them via their corporate contact form. Although the contact form generated an automated acknowledgement, there was still no substantive response from China Citizen Watch and the data continued to leak.
On December 2, I tried contacting the host, upmcn.com, and sent a third notification to China Citizen Watch.
Neither responded at all.
On December 3, I contacted Citizen UK in the hope that they would be able to ensure that Citizen Watch China would respond. Over the course of the next week, we would go back and forth, but nothing got done, and on December 10th, they informed me that they had forwarded the information to the American CTO.
The data continued to leak.
By six days after contacting Citizen Watch UK, nothing had changed.
China Citizen Watch was sent a 4th email notification on December 11…
and a 5th one on December 17….
In addition to banging my head against the wall by repeatedly trying to notify them via email, I also attempted to contact them via their corporate webform request and also via their web chat. Their web chat appeared to be mostly offline, even when it was supposed to be online.
I also tried making contact via LinkedIn to various higher level Citizen staff from the U.S., Canada, and Japan. Not one of the following individuals responded at all:
William Parizeau Fillion
Marketing & IT Manager at Citizen Watch
Ottawa, Canada Area
Nancy Garcia, SHRM-CP
Senior Human Resources Manager at Citizen Watch America
Kevin Kaye
President at Citizen Watch Company of Canada, Ltd.
Regina Fiedel
Vice President Marketing at Citizen Watch America
Glenn Parker
Vice President, Human Resources at Citizen Watch America
12th
Trish Keller
Chief Technology Officer of Citizen Watch America
On December 18, I sent a 6th email notification, this time using an email address I had discovered in a sample of their data. That seemed to get their attention, and in reply, they asked for my IP address and the time at which I had accessed their RSYNC. I suddenly found myself concerned that they might try to shoot the messenger.
In sum:
- it took 25 days from discovery to seeing the system secured and taken offline
- it took 6 notification emails to more than 20 different people
- It took email, LinkedIn messages, webchat messages, and Twitter messages.
- It took a call to the New York corporate headquarters Press Office by DataBreaches.net, who left a detailed message but got no return phone call.
It should never have been that difficult. A major corporation like Citizen Watch should have better incident response than this. China Citizen Watch may not be directly at fault as they are using a service called upmcn.com who explains what they do for Citizen:
“Citizen’s disaster recovery plan includes two parts: the local data center and the remote (the IDC center) disaster recovery center. The application of the local data center is backed up and then backed up to the remote disaster recovery center. The local data center has 8 virtual machines and 3 separate servers, and another backup server. Applications deployed on the virtual machine include: file server, mail server, anti-virus software server, domain control, vcenter, instant messaging server, print management platform system, after-sales telephone call center system, after-sales service order management system, using virtual machine backup mode Scheduled backups. The three separate servers are the retail terminal management system server, the file server for the e-commerce department, and the SBO ERP server.
The volume CDP is used to perform real-time backup of the database, files, and operating system. Local data backup prevents small disasters, and offsite data backup prevents major disasters, comprehensively protects data security, and records bits and pieces.”
Using remote cloud backup services is becoming a much more common thing these days, specially for big companies like China Citizen Watch who have hundreds of servers and systems linked and working together to make their company run. Both the Citizen and Bulova-branded watches are very popular watches on the market. How can they not be checking for security notifications or responding to them?
Sadly, what happened here is nothing new or different. During this same period, I was also notifying Adidas, SonyLIV, and a number of hosting companies who also left RSYNC backups open or other services unsecured. Thankfully, it didn’t take as many notifications to get responses from them.
But if a big company like China Citizen Watch can’t get incident response right, doesn’t acknowledge notifications, and doesn’t follow up properly, well, that doesn’t bode well for smaller entities.
What do I mean by “doesn’t follow up properly,” you wonder? Although I was initially happy to see the RSYNC seemingly locked down, it took only one minute to realize that there were still other ways to access the data. Yes, the original access was secured, however there were still other ways to access the data via http-based login interfaces.
So I sent them yet one more notification to alert them they still had access via http and that they needed to restrict the direct and domain access as various domains was attached to the IP address, all of which showed the same login interface.
This time, they responded promptly, thanking me for my multiple emails and saying that they “have taken steps.”
Research and reporting by Lee Johnstone. Editing and additional reporting by DataBreaches.net