When they discovered more than 42,000 patient records and millions of patient clinical notes exposed on a misconfigured rsync backup, researchers at UpGuard responsibly set out to notify the entity to secure their data. It turned out to be a Herculean task that would take almost two months and multiple entities to get the job done. That shouldn’t be.
It’s been a while since Chris Vickery, now the Director of Cyber Risk Research at UpGuard, and this blogger teamed up to try to get an entity notified. Recently, though, he contacted me for assistance. A medical group in my area was leaking patient data from an rsync device left open on port 873, but he was having trouble reaching them, he said.
The data in the leaky backup appeared to belong to Cohen, Bergman, Klepper & Romano in Huntington, New York, Chris told me. The group had no web site, and there were no publicly available email addresses for them – only the address of their office and a phone number. The doctors appeared to be affiliated with – but not necessarily employed by – Huntington Hospital. Foolishly, my first thought was, “How hard can this be? Heck, I can just jump in my car and drive over to their office if need be.” With the benefit of hindsight, I probably should have driven over there as it would have saved me a lot of time and aggravation.
According to Chris, the exposed data included patient names, date of birth, address, and phone number, as well as Social Security numbers, some health insurance information, and more than 3 million clinical notes. In one table alone, there were more than 42,000 patient records rows. Personal information of the doctors and office staff was also in the backup.
UpGuard had tried to email the medical group on February 12 (and again on February 19), he said, using email addresses they found in the leaking backup device, but they never got any response. They had also tried to call the practice’s office on February 12, only to have an employee hang up on them.
Read UpGuard’s report on this incident on their site.
Hoping that Huntington Hospital might reach out to the doctors to alert them to the leaky backup, UpGuard had also called Huntington Hospital administrative staff on February 21 and February 23, only to get no call back despite having been assured of a call back both times.
For more than one month, UpGuard had made multiple attempts to reach the doctors via multiple methods, all to no avail. And that’s why they had contacted this site.
The more I looked at what UpGuard had sent me, the more concerned I became. The data were on an Optimum (Optonline.net) IP address, and Optimum is very clear in their privacy policy that they are not HIPAA-compliant.
So what was all that unencrypted patient data doing on Optonline?
And for how long had it been exposed that way? From the limited data UpGuard shared with me, it appeared that the backup may have been last updated in July, 2015. And from my research, it appeared that the Cohen in Cohen, Bergman, Klepper and Romano may have retired or left the practice at some point. Was this an abandoned backup? Had the practice done a risk assessment, and if so, when was the last time they inventoried this? And when was the last time they checked its security?
This was turning out to be much messier than I had imagined.
Replicating some of UpGuard’s steps, I sent notification emails to the same email addresses that UpGuard had tried. My emails to those addresses did not bounce back, but I also got no response and the data remained unsecured.
I decided to call Northwell Health System’s corporate compliance office. It was Saturday afternoon, so not surprisingly, my call went to voicemail. Note that I did not call Northwell because I thought they had any responsibility for the exposed data. I called them because I figured if the doctors got a phone call from the major health system they were providers for, they might not hang up on them. I left a message on Northwell’s voicemail and prepared for a long wait.
Because Optimum’s Twitter team appeared to be around fairly regularly on Twitter, I also reached out to them that Saturday. I sent them a DM:
If I give you an IP address and it’s an Optonline IP addy, can you contact your customer and tell them they’re leaking sensitive data and they need to secure their rsync device. And keep this in DM, please so no one figures out where to go to find highly sensitive data.
On Sunday, Optimum’s Twitter team answered me:
Hello! That is not something we would be able to do. The customer would have to contact us directly for us to look into their account.
Well, d’oh…you can see the problem there, right? I gave up on Optimum for the time being.
By now it was Sunday, and I was still mulling over how to get through to the medical group. I decided to call the medical practice, even though they were closed. I figured I’d get their answering service, and maybe the answering service would actually call a doctor and get them to call me back. I had done that successfully before, so it was worth a shot, I figured. I called, got the answering service, and explained that I needed to speak to the doctor(s) about a fairly urgent data security breach exposing tens of thousands of patients’ records. The answering service employee told me that since this was not a medical emergency, it would have to wait until the next day and that he would pass the message along.
A few hours later, my phone rang. It was someone from Northwell Health corporate compliance. I gave her some more details and history of notification attempts, and she assured me that they would try to help. Score one for them.
The next day, when Chris checked, the device had been secured. I was relieved, but I didn’t know what had worked to get through to the doctor(s). Was it Northwell’s intervention or the call to the answering service? What had worked? I’d like to know for the future.
A few days ago, I got another phone call from Northwell Health. They were following up with me, and gave me some additional details. It seems that after I contacted them, they reached out to administration at Huntington Hospital, and a senior physician from Huntington Hospital in the same specialty as the medical group then called the medical group to tell them about the leaky device. And hallelujah, they had not hung up on that doctor.
Huge kudos to Northwell Health for their response to a weekend request for help securing data that was not even their data. I am not naming individual employees of Huntington Hospital or Northwell because I get the sense that a number of people were contacted and involved to help finally deal with this situation. DataBreaches.net thanks them all.
As to the medical group who appeared to be responsible for the device? Well, I still haven’t spoken with them as they never called me back even after getting that message from their answering service. So I have no answers for you as to how long that device was exposed, who was responsible for it, how many times it might have been accessed or downloaded, and why it was on Optonline which is not HIPAA-compliant.
And because I have no idea if they intend to notify any patients or regulators or what they will do, I’ve decided to file a complaint about this incident with HHS and ask them to open an investigation into what happened. There really needs to be serious consequences for covered entities who do not have a system in place to receive and to respond to notifications of security concerns. And that’s apart from the issue of why these data were on Optimum, and whether they had lost track of a device with patient records.
But for now, thus endeth this edition of Adventures in Notifications. And Chris: you now owe me a coffee when I see you.
Update: A complaint/request for investigation has now been filed with HHS. I do hope they investigate but realize that they do not always pursue what I wish they would pursue.