PM861 communication lost

ABB team,

we have 30 redundant PM861 on our 30 well heads and are facing below issue on regular basis.

Controller is healthy and suddenly communication drops (controller becomes inaccessible on ABB OPC utility)

Scenario # 1 – Control builder unable to go online and HMI communication lost, but ping to controller is available. I/O scanning and logic execution is normal. Controller has to be downloaded again to make communication healthy. Log attached.

Scenario # 2 – Controller becomes inaccessible at all, no ping, no I/O scanning, no logic execution, no download availability. Controller has to give power cycle to make it available for download.

Thanks & Regards,
Rizwan ul Hassan.

 Log.zip


By Engr. Rizwan Hassan on 9/10/2017 | Like (0) | Report

In first scenario, system status viewer shows a warning of network storm and while downloading it shows download for the first time.


By Engr. Rizwan Hassan on 9/10/2017 | Like (0) | Report

for 2nd scenario controller data is not available yet due to remote sites.


Add New Comment


Engr. Rizwan Hassan   

asked 2 months ago
Closed



Best Answer

1

I recommend that you check the Ethernet connections.

Your comment about observing storm warning concur with the controller log(s). There are communication errors with peer controllers and hardware warnings from both Ethernet ports (0.1 and 0.2).

The AC 800M is designed to shut down if subjected to too high load, e.g. due to incoming Ethernet telegrams. The Network Storm Protection (NSP) might not be fast enough to save the day.

The close (in time) messages from primary and secondary Ethernet ports may indicate an unfavorable installation or wiring. ABB recommend having primary and secondary Ethernet connections fully separated (do not use VLAN or other muxing equipment resulting in a blend of the traffic in some common media/fiber/radio/etc).

If you need further assistance, please contact your regional ABB support or sales representative.

E 1979-12-31 00:00:07.280 RNRP Config error: Socket send queue full to 172.17.80.177. Short sendPeriod?
E 1979-12-31 00:00:07.766 RNRP Config error: Socket send queue full to 172.16.80.165. Short sendPeriod?
...

E 1979-12-31 00:00:06.770 RNRP Config error: Socket send queue full to 172.16.80.170. Short sendPeriod?
E 1979-12-31 00:00:07.266 RNRP Config error: Socket send queue full to 172.17.80.178. Short sendPeriod?
...

W 2017-09-09 01:01:13.045 On Unit= 0.1 HWError Contro~_RTU_12 0000 See HWTree Error 16#40020000 16#00000000
W 2017-09-09 01:01:13.046 On Unit= 0.2 HWError Contro~_RTU_12 0000 See HWTree Error 16#40020000 16#00000000

Stefan Stromqvist   

answered 2 months ago


 


By Engr. Rizwan Hassan on 9/14/2017 | Like (0) | Report

Thanks Stefan for the response. I would like to have your recommendation on use of stack switches for ABB network. Our Pri & Sec networks are combined on radio but we are using stack switches on connectivity server side to avoid network storm on main DCS and ESD controllers.


By Stefan Stromqvist on 9/14/2017 | Like (0) | Report

If possible, ABB recommend full isolation between primary and secondary network paths. Then I would be surprised if both paths fail simultaneously.

I cannot comment on the stacking - I don't know the configuration, make and model.

As you already have experienced, the present network does not provide enough separation and has at least during one occurrence acted as "one network" and caused loss of communication.

Storm protection is recommended. Our ABB NE800 equipment is preset with a storm filter blocking at 2 mbit/second of broadcast, multicast and unknown unicast. These settings might need further adjustments depending on site environment. Storm protection is no replacement for isolation, but may reduce the adverse effects a sustained storm may bring.


By Engr. Rizwan Hassan on 9/14/2017 | Like (0) | Report

Thanks a lot Stefan, We have Isolated RTU wells network from our DCS / ESD network because radio bridging was causing network storm and recently we faced ESD controller halt as well. This is due to a design fault by radio team that they didn't separate networks. Thanks for you support. Have a good day.


Add New Comment


Answers

1

I recommend that you check the Ethernet connections.

Your comment about observing storm warning concur with the controller log(s). There are communication errors with peer controllers and hardware warnings from both Ethernet ports (0.1 and 0.2).

The AC 800M is designed to shut down if subjected to too high load, e.g. due to incoming Ethernet telegrams. The Network Storm Protection (NSP) might not be fast enough to save the day.

The close (in time) messages from primary and secondary Ethernet ports may indicate an unfavorable installation or wiring. ABB recommend having primary and secondary Ethernet connections fully separated (do not use VLAN or other muxing equipment resulting in a blend of the traffic in some common media/fiber/radio/etc).

If you need further assistance, please contact your regional ABB support or sales representative.

E 1979-12-31 00:00:07.280 RNRP Config error: Socket send queue full to 172.17.80.177. Short sendPeriod?
E 1979-12-31 00:00:07.766 RNRP Config error: Socket send queue full to 172.16.80.165. Short sendPeriod?
...

E 1979-12-31 00:00:06.770 RNRP Config error: Socket send queue full to 172.16.80.170. Short sendPeriod?
E 1979-12-31 00:00:07.266 RNRP Config error: Socket send queue full to 172.17.80.178. Short sendPeriod?
...

W 2017-09-09 01:01:13.045 On Unit= 0.1 HWError Contro~_RTU_12 0000 See HWTree Error 16#40020000 16#00000000
W 2017-09-09 01:01:13.046 On Unit= 0.2 HWError Contro~_RTU_12 0000 See HWTree Error 16#40020000 16#00000000

Stefan Stromqvist   

answered 2 months ago


 


By Engr. Rizwan Hassan on 9/14/2017 | Like (0) | Report

Thanks Stefan for the response. I would like to have your recommendation on use of stack switches for ABB network. Our Pri & Sec networks are combined on radio but we are using stack switches on connectivity server side to avoid network storm on main DCS and ESD controllers.


By Stefan Stromqvist on 9/14/2017 | Like (0) | Report

If possible, ABB recommend full isolation between primary and secondary network paths. Then I would be surprised if both paths fail simultaneously.

I cannot comment on the stacking - I don't know the configuration, make and model.

As you already have experienced, the present network does not provide enough separation and has at least during one occurrence acted as "one network" and caused loss of communication.

Storm protection is recommended. Our ABB NE800 equipment is preset with a storm filter blocking at 2 mbit/second of broadcast, multicast and unknown unicast. These settings might need further adjustments depending on site environment. Storm protection is no replacement for isolation, but may reduce the adverse effects a sustained storm may bring.


By Engr. Rizwan Hassan on 9/14/2017 | Like (0) | Report

Thanks a lot Stefan, We have Isolated RTU wells network from our DCS / ESD network because radio bridging was causing network storm and recently we faced ESD controller halt as well. This is due to a design fault by radio team that they didn't separate networks. Thanks for you support. Have a good day.


Add New Comment



Get weekly AKS updates


Partner Exclusive Webinars

 

> – Login to the partner portal to register



Points Redemption Program - Redeem your points for ABB training, Bluetooth speakers and mugs. Terms and conditions >


Earn points when you refer a friend
AKS Referral Program is "Live" - Learn more



AKS Experts


avatar
Ask Flavio Mussolin   

Rank: 4

ABB AVP, Senior Electronic and Automation Engineer with over 30 years of experience in the field of process automation automotive, pharmaceutical, hollow glass, steel and rolling.


avatar
Ask kstoilov   

Rank: 15

Control System Engineer: 800xA, Compact 800, AC500, AC31-50, Advant Master, Simatic, AC&DC Drives 11 years worked for ABB - Control Systems


avatar
Ask nikismith   

Rank: 266

I have been a part of the Recording & Control Factory team for 17 years in total, having spent a number of years as a repairs technician withi the manufacturing department, but being in my current role for 9 years now.


avatar
Ask MMM   

Rank: 5

ABB PA CT Technical Support


avatar
Ask Harsha.D   

Rank: 6

Tech.Support,software and commisioning engineer with Proficient knowledge in 800xA and its products, RNRP,Networking in general.


avatar
Ask Govindaraj   

Rank: 10

Working in ABB India Operation Center. Have Project engineering and commissioning experience in ABB 800xA, Freelance, AC500.


avatar
Ask Stefan Stromqvist   

Rank: 1

I joined ABB in the year of 1994 and has since 1999 been working as a Service & Support Engineer at BU Control Technologies in Västerås, Sweden. My areas of expertise are: 800xA Base, 800xA for Advant Master, Information Management, operating systems, RNRP and Ethernet comms/networking in general.


avatar
Ask Rob Lyon   

Rank: 3

info@lymac.co.nz I'm an independent DCS software and commissioning engineer with extensive experience in 800xA and other ABB products. I've worked in almost every corner of the world and still haven't seen it all.


avatar
Ask Dieter Henkel   

Rank: 23


avatar
Ask Sumit Gargav   

Rank: 2

I have worked with Freelance in majority, with 800xA FD and Melody system partly. Also had opportunity to work with Protocols - HART,Profibus,FF & Modbus.