Wednesday 3 June 2020

Voice Call Bot : Bashing NHS England's appointment booking system

What do you do when you have to call a number for availing a service, but the number being called is so contended that your call never goes through?


The COVID19 crisis has exposed the limitations of several such systems. Case in point, England's NHS (National Health Service). To get a doctor's appointment you have to call them. There is no provision for getting the appointment online. So, if you have to get an appointment for, say getting blood sugar level tested, you will be required to call them, but since the health services are already overwhelmed by COVID19 crisis, and there are thousands of people trying to call them, you, simply won't be able to make the call. Either the network would be busy, or there would be other such problem.

Similar was (and probably still is) the situation in USA. COVID19 has caused unprecedented level of unemployment, leading to a surge in calls to unemployment offices for people wanting to avail unemployment benefits. There have been reports of people calling them hundreds, and thousands of time without success.

The Bot

While people get frustrated, computers are good at doing these mundane, and repetitive tasks.
Our aim is to make a bot which calls NHS on our behalf, and when it is successful, somehow connects us to it.

Here is a state diagram explaining the process.


The bot will keep on trying to call NHS till someone answers. It will then call the user on the given number, and will connect the two calls. The user will be able to talk on the call to NHS as on a normal call.

Prerequisites 

  1. Twilio subscription [Link]
    Twilio has a rich set of telephony API. The downside is that to be able to make calls to real numbers, you will have to convert your account into a normal account if it is a trial account right now. For this you will have to load at least 20 USD in your account. This money will be utilized for making calls.
  2. A server to run the bot. The server has to have a public IP. I purchased a VPS from Digital Ocean for 5 USD a month. You can find cheaper VPS on lowendbox.com. Alternatively you can also use your local machine, however you may have to configure NAT forwarding if you want to make it accessible from public. See configuring VPS section below for more options.

Configuring Twilio 

When you load at least 20 USD in your Twilio account, your account will non longer be in trial, and you will be able to make calls to real numbers. In trial mode, you are only allowed to call Twilio provided numbers. It is a good idea to use these numbers to test your code before investing money.

You also need a number which will be used for caller identification (Caller ID) when making calls from Twilio.
You can either register the number you already have, or buy a new one from Twilio. The former will not cost you anything extra.
For registering an existing number, go to https://www.twilio.com/console/phone-numbers/verified.
For buying a new number you can go to https://www.twilio.com/console/phone-numbers/search.
I used my existing number for the purpose.

Configuring VPS

You will need VPS for 2 reasons.
1. Run script which will invoke Twilio API to make the call.
2. Host an XML file which will tell Twilio what to do once someone answers the call. This will be in TwiML, Twilio Markup language. 
3. Get status feedbacks. As your call progresses through various stages, Twilio will notify you by sending this information to a configured URL. In Twilio's terminology, this is called StatusCallbackEvent.




This is the timeline of a typical Twilio call (image taken from https://www.twilio.com). For each of these stages, when it is reached, the given URL will received information. This is how your code will come to know the status of your call. The call state which is of most interest to us is answered. This is the point which indicates that NHS has picked up our call, and Twilio should call the user on the given mobile number, and then connect the 2 calls.

To save money, you can run the script on your local machine, and get call status feedback on a php script hosted on a website for free. I gave https://infinityfree.net a shot for free hosting, but it didn't work out for me, as the site sets some cookies automatically, and the twilio API wasn't able to handle them, maybe there is a way, but I didn't explore further.

The VPS I purchased had Ubuntu 18.04, with apache2, lib-apache2-mod-php, python, python-pip packages installed. I also had to install Twilio python package from pip.


pip install twilio

Writing scripts

I wrote 2 scripts. One in python, the other in PHP. While the Python script calls Twilio API, the PHP script provides an endpoint where Twilio can send call status events.

The python script

 
from twilio.rest import Client
import os.path
import time
import pdb

# Your Account Sid and Auth Token from twilio.com/console
# DANGER! This is insecure. See http://twil.io/secure
account_sid = 'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth_token = 'YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY'
client = Client(account_sid, auth_token)

filePath='/var/www/html/callid.txt'

while True:
    call = client.calls.create(method='GET',  status_callback='http://xxxx.com/yyy.php', status_callback_event=['initiated', 'answered', 'completed'], status_callback_method='POST', url='http://xxxx.com/response.xml', to='+44xxxxxxxxxx', from_='+91xxxxxxxxxx')
    print(call.sid)
    while True:
        if os.path.isfile(filePath):
            print("file exists")
        else:
            print("file does not exist. sleep 5s ...")
            time.sleep(5)
            continue
        try:
            print("sleep 5s ...")
            time.sleep(5)
            with open(filePath) as f:
                fline = f.readline()
                print(fline)
                if(len(fline) == 0):
                    continue
                ftokens=fline.split(' ')
                fcid    =   ftokens[0]
                fstatus =   ftokens[1]
                if str(call.sid).upper() == fcid.upper():
                    print("cid match")
                    if(fstatus.lower() in ["busy", "canceled", "completed", "failed", "no-answer"]):
                        print("call ended. dialing again in 60s ...")
                        time.sleep(60)
                        break
        except IOError:
            print("file open error")

You can get account_sid, and auth_token from twilio console (https://www.twilio.com/console).
Additionally, you will have to configure following parts in the script.
  1. url='http://xxxx.com/response.xml'
    This this the XML file which has Directions written in TwiML.
    Once a call is answered Twilio reads this file to decide the
    actions which have to be taken. For example you can instruct
    Twilio to say something, play an audio file, etc. In our XML file we instruct Twilio to call the user.
    <Response>
            <Say voice="alice">Hello</Say>
            <Dial>+91xxxxxxxxxx</Dial>
    </Response>
    
    The XML file tells twilio to say hello once call is picked up,
    and then dial the given number. The given number will be
    automatically joined to the current call.
      
    

  2. status_callback='http://xxxx.com/yyy.php'
    This is the URL on which Twilio sends call status data as call progresses. Once this URL receives information that a call has ended, our python script tries again.
  3. to='+44xxxxxxxxxx', from_='+91xxxxxxxxxx'
    These are the to, and from numbers. In my case to was NHS's number, and from was my number which I had registered with Twilio for caller ID.

The PHP script

The PHP script is meant to receive call status information from Twilio. The PHP, and Python scripts work together to decide when a call failed, so that we can try calling again.

<?php
$req_dump = print_r($_REQUEST, TRUE);
$fp = fopen('request.log', 'a');
fwrite($fp, $req_dump);
fwrite($fp, "#\n");
fclose($fp);

$fp2 = fopen('callid.txt', 'w');
fwrite($fp2, $_REQUEST['CallSid']);
fwrite($fp2, ' ');
fwrite($fp2, $_REQUEST['CallStatus']);
//fwrite($fp2, '\n');
fclose($fp2);

?>
<!DOCTYPE html>
<html>
</html>

This PHP script writes to a file named callid.txt which is monitored by the Python script.
When Python script makes a call it gets an ID identifying the call (call.sid). When it sees that the
status of the call it just placed is one of "busy", "canceled", "completed", "failed", "no-answer", it realises that call has ended, and it is time to try again.

One thing to note here is that the python script will keep calling even when call is disconnected after being answered. Since the user will get a call as soon as call is answered, the user can then manually kill the script, so I didn't feel like investing time on this.

Running the bot

Put the PHP script in apache's directory, and create two files, callid.txt, and request.log in it, with read/write permissions given so that both PHP, and Python scripts are able to access them.

Configure the python script as explained above, and then just run it.
Make sure to kill it once you get a call on your phone.

Parting thoughts

As of now it has been 6 hours since I started the script. Probably the NHS number I have is unattended.
The script does work, as before running it on NHS number I tested it with several US, UK, and Indian numbers.

My initial Idea was to script it using Tasker on Android, but Android sufferes from a major drawback. There is no public API which would tell us if the call has been answered. The dialer on Android starts the call timer only when the call gets answered, this means the OS is aware, but has not exposed any API for it. See the comment on PreciseCallState hidden Android API on the following stack overflow page - https://stackoverflow.com/questions/9684866/how-to-detect-when-phone-is-answered-or-rejected

References

  1. Samples and tutorials on making call using Twilio API - https://www.twilio.com/docs/voice/make-calls?code-sample=code-make-a-call-and-monitor-progress-events&code-language=PHP&code-sdk-version=6.x#specify-a-recordingstatuscallback
  2. Call recording API - https://www.twilio.com/docs/voice/api/call-resource#fetch-a-call-resource 
  3. Twilio conference calls (Note that we didn't use this API in our bot) - https://www.twilio.com/docs/voice/tutorials/how-to-create-conference-calls-python