The ICT service

Current Firewall and Network Issues

As at Tuesday 24 January 2017

We are sure you will all be aware by now we have been experiencing intermittent connectivity issues with the Internet connection shared by all Cambridgeshire schools since the afternoon of Tuesday 17 January 2017.
 
We have so far identified this to be a fault with our core Firewalls.  These are devices that act as a barrier from the Internet protecting schools from unauthorised access. We have dedicated our most senior technical resources and are in constant communication with the manufacturer who is supporting us.
 
As the root cause of this fault is not obvious to us or the manufacturers, we have taken several steps to mitigate the impact it is having on our services and schools. One such step was to reduce the overhead on the Firewalls by turning off some important, but not daily essential services. This has had the desired effect and since Thursday at 9.40am there have been a small number of outages compared to Tuesday and Wednesday when it was every 10 minutes.
 
Diagnosis of this type of problem is predominantly carried out by reviewing the configuration and logs for the Firewalls. Due to the scale of our Firewall implementation and complexity of this issue, only the Firewall manufacturer can pin point the data within the logs to identify the root cause.
 
Because of the workarounds we have implemented there was some time between each failure and because of the added time for the manufacturer to subsequently analyse the logs, it has turned into a very drawn-out process to find the resolution.  It has since been necessary for us to switch services back on to produce the fault more frequently to give further troubleshooting information.  
 
We turned the services back on in the evening of Monday 23 January at 5.45pm, at which point the service was disrupted for 5 to 10 minutes. We then lined up a dedicated resource from our Firewall provider to be available during peak customers’ login times, on Tuesday 24 January, to view the logs and fault in real time. Unfortunately, this meant that there was an increased level of disruption while we gathered this information. We approximated that we should have enough logging information by 11:30am.
 
We apologise for the increased disruption but we believe it is essential to finding the root cause and to resolve the issue.
 
As soon as we have enough information we will attempt to return the Firewall to a state where it minimises the impact to schools if an immediate fix is not found.
 
We will continue to provide updates via our normal communication channels in instances such as these.
 
As a reminder, you can get updates here: 
 
Status website - http://status.theictservice.org.uk – please note the availability statistics on the site are a 5 minute checks, if there is a short or intermittent outage they may not be captured.
 
Twitter - https://twitter.com/ICTServiceTech - this feed is also displayed on the side of the status website in viewed in a desktop browser and on our website www.theictservice.org.uk.
 
Facebook - https://www.facebook.com/ICTServiceTech
 
The ICT Service Helpline - call us on 0300 300 0000.  
When we experience issues like this we add the latest information as a message before you are put through to the team.
 
I do hope you will agree that this is very much out of the ordinary in terms of the level of service we provide and I thank you for your patience.  I can assure you this is our highest priority and working to return the service to normal levels as soon as possible.
 

Customer Request/Incident

We invite our customers to use our online form to quickly let us know if you have a request or an incident to report.

Submit details


Your Basket

  • No products selected

enquire

Add products to your basket to enquire about multiple services


what people say…

  • I just wanted to let you know how happy we have been with a recent server installation at school. Great service and so prompt.

    Head, Abbots Ripton C of E Primary School

  • We have used The ICT Service now for many years, and have the highest praise for all the departments we deal with (we couldn’t live without our ICT technician or the helpdesk!).

    Weatheralls Primary School

  • Always helpful to guide me through the troubled waters of the Post 16 Census! I don't know how I would manage without them.

    Christine Bradshaw, St Ivo School

  • Our Technician always goes the extra mile for us. He understands our school needs and limitations. He is considered to be one of our school team.

    Primary School Headteacher

  • ...really helplful during a very stressful time...

    Secondary School ICT Technician

  • Very helpful, I like the way that The ICT Service can come on to the system remotely it saves so much time!

    Chris Cater, Cottenham Primary School

  • I always find the technical team extremely helpful and knowledgeable and very patient with whatever question you ask!

    Emily Lewney, Bar Hill Community Primary School

  • Really clear, easy to follow course with some great ideas.

    Sarah Dodsworth, Wisbech St Mary Primary School

  • I attended a truly fantastic course which has really opened my eyes to how I can use SIMS to support my data analysis.

    Yasmin Trace, Hatton Park Primary School

  • I have been in this job over 10 years and find the team a comfort at times of stress over jobs that have gone wrong. They are a credit to the service.

    Rebecca Mack, Business Manager, The Fields CC

  • Lorraine on The ICT Service Helpline was very patient and friendly and seemed genuinely pleased to be helping me.

    Sheena Marsh, Sawtry Infant School

  • The course answered all my questions but went far beyond my expectations. Very good value for money. I also appreciate the handbook.

    Margaret Carpenter, Longsands Academy

  • All courses that I have attended are so helpful and are explained in terms that I can understand and I am not afraid to ask questions.

    Sarah Walker, Thorndown Primary School

  • The Helpline are always great when I ring and try to sort the problem there and then at all times if possible. Great service.

    Paula Beeby, Coates Primary School

  • Everyone I deal with on the Helpline is helpful and patient especially understanding the pressures we're under. It's lovely to know we have an efficient team available when needed.

    Lynn Hilling, The Vine Primary School

  • I was given extremely clear and understandable instructions to help me find my way through difficulties.

    Sue Bardell, LLDD Team, CCC

  • The trainers was very happy to answer all questions so that we are able to use the system in the most effective way for our school

    Alison Rowbotham, The Weatheralls Primary School

  • Thank you. The e-safety session was one of the best uses of time that I have had for a long while.

    E-Safety Course

  • I found the whole session incredibly useful. You gave us lots of useful resources to take back to school and even more to think about.

    E-Safety Course

  • Emma always explains what you need to do very clearly and is extremely helpful - Thank you

    Sue Scott, Alderman Payne Primary School

  • I don't know what I would do without The ICT helpline. I have always found them to be friendly and supportive whenever I have called

    Primary School Secretary

latest tweet

  • Scheduled downtime is complete, all services are back online. For support, contact The ICT Service Helpline on 0300 300 0000 #ICTDowntime

    1 week


info@theictservice.org.uk

Tel:0300 300 0000